tcb@breakable

# PACE-LM: Prompting and Augmentation for Calibrated Confidence Estimation
with GPT-4 in Cloud Incident Root Cause Analysis

Shizhuo Dylan Zhang<EMAIL_ADDRESS>University of Illinois Urbana-
ChampaignChampaignIllinoisUSA ,  Xuchao Zhang<EMAIL_ADDRESS>MicrosoftRedmondWashingtonUSA ,  Chetan Bansal<EMAIL_ADDRESS>MicrosoftRedmondWashingtonUSA ,  Pedro Las-Casas<EMAIL_ADDRESS>MicrosoftRedmondWashingtonUSA ,  Rodrigo Fonseca
<EMAIL_ADDRESS>MicrosoftRedmondWashingtonUSA and  Saravan
Rajmohan<EMAIL_ADDRESS>MicrosoftRedmondWashingtonUSA

###### Abstract.

Major cloud providers have employed advanced AI-based solutions like large
language models to aid humans in identifying the root causes of cloud
incidents. Despite the growing prevalence of AI-driven assistants in the root
cause analysis process, their effectiveness in assisting on-call engineers is
constrained by low accuracy due to the intrinsic difficulty of the task, a
propensity for LLM-based approaches to hallucinate, and difficulties in
distinguishing these well-disguised hallucinations. To address this challenge,
we propose to perform confidence estimation for the predictions to help on-
call engineers make decisions on whether to adopt the model prediction.
Considering the black-box nature of many LLM-based root cause predictors,
fine-tuning or temperature-scaling-based approaches are inapplicable. We
therefore design an innovative confidence estimation framework based on
prompting retrieval-augmented large language models (LLMs) that demand a
minimal amount of information from the root cause predictor. This approach
consists of two scoring phases: the LLM-based confidence estimator first
evaluates its confidence in making judgments in the face of the current
incident that reflects its “grounded-ness” level in reference data, then rates
the root cause prediction based on historical references. An optimization step
combines these two scores for a final confidence assignment. We show that our
method is able to produce calibrated confidence estimates for predicted root
causes, validate the usefulness of retrieved historical data and the prompting
strategy as well as the generalizability across different root cause
prediction models. Our study takes an important move towards reliably and
effectively embedding LLMs into cloud incident management systems.

††copyright: none

##  1\. Introduction

In the past decade, the IT industry has been progressively adopting cloud
platforms as their preferred method for deploying applications and services.
Within these expansive cloud services, incidents like unexpected interruptions
or performance declines can substantially harm customer satisfaction, leading
to revenue loss and a decrease in customer confidence. Root Cause Analysis
(RCA), as a critical task in the incident diagnosis process, aims to uncover
the underlying causes of issues and implement corrective measures to prevent
future occurrences. Identifying the incident’s root cause is crucial for
reducing service downtime, mitigating customer impact, and minimizing manual
effort.

Recently, Large Language Models (LLMs) have demonstrated their proficiency in
incident diagnosis tasks, including root cause analysis. They play a crucial
role in gathering information from historical data and various sources,
providing invaluable recommendations to on-call engineers (OCEs) when
addressing intricate cloud incidents. While LLMs can deliver promising
outcomes in numerous situations, they still generate a significant number of
incorrect recommendations. These inaccuracies have the potential to misguide
OCEs into making erroneous decisions, leading to a waste of their diagnostic
efforts. As shown in the example depicted in Figure 1, there are situations
where LLMs can provide invaluable insights, reducing the need for exhaustive
report analysis, delving into past incidents, or data gathering. However, in
other instances, they may appear vague, lacking in detail, or even misleading.
Moreover, LLM-based approaches are susceptible to generating hallucinations.
What’s concerning is that recognizing these hallucinations can be a
challenging task, making it difficult for engineers to determine their
authenticity. These issues significantly impede the incorporation of AI tools
into the cloud incident management process, as erroneous predictions can lead
engineers astray, resulting in significant time wastage and potentially
serious consequences for both the service provider and its users.

In order to address the gap in the practical application of AI-assisted Root
Cause Analysis (RCA), we propose the solution to estimate the confidence level
associated with each predicted root cause. By establishing a reliable
confidence estimation mechanism, on-call engineers can make more informed
decisions. This, in turn, enhances the efficiency and effectiveness of
incident resolution while reducing the costs incurred by delays caused by
inaccurate root cause recommendations from the model. Moreover, preventing the
adoption of incorrect recommendations not only improves the user experience by
minimizing disruptions but also reduces the on-call effort required to rectify
such issues. By gaining a deeper understanding of the reliability of AI-based
model output, organizations can confidently integrate these tools into their
operational workflows, ensuring seamless and reliable service delivery in
cloud environments.

Figure 1. An example cloud incident with a series of predicted root causes by
different root cause predictor models. This illustrative example shows that
the model predictions are of mixed quality even for the same incident. The
strength of green (red) color for model-predicted root causes indicates how
likely they are correct (wrong). It is apparent that if such information can
be obtained beforehand, we could significantly assist on-call engineers in
making decisions on how much should they read through and rely on each
recommendation.

However, developing a calibration procedure for root-cause analysis is itself
challenging, encompassing a range of factors that complicate the process.
First, the root cause predictors might vary across time and services. Our
solution needs to be general enough and decoupled from the underlying model.
Second, the access to the model can be limited. For example, the logits and
probabilities of GPT-3.5-Turbo and GPT-4 models are not disclosed, making it
difficult to apply fine-tuning-based approaches or traditional post-hoc (Jiang
et al., 2021) methods like temperature scaling to directly obtain confidence
scores from the root cause predictor. Further, the open-ended nature of the
problem adds to the challenge. Compared with categorical tasks like multiple-
choice or classification, calibration for open-ended question-answering tasks
is more complex in nature due to the unconstrained output space, and
difficulty in defining direct supervision signals. This paper aims to make
progress in resolving the critical challenge of confidence calibration in
domain-specific applications, with a focus on root cause analysis of cloud
incidents. The calibration problem involves evaluating the plausibility and
reliability of the outputs generated by AI-empowered cloud incident root cause
analysis systems, which requires both comprehension of the incident and
predicted root cause, and the critical analysis of the prediction. To this
end, we adopt a large language model (LLM)-based solution to perform
confidence estimation.

Nevertheless, it is worth emphasizing that LLMs are primarily trained on a
general-domain corpus from the internet, while the handling of cloud incidents
and root causes requires domain knowledge. There is no surface-level
connection between an incident description to its root cause, and the model
therefore is not able to deduce to the conclusion on the extent to which a
root cause is valid by utilizing only its general-domain knowledge and
commonsense logical reasoning capacities. To bridge this gap and enhance the
performance of the confidence estimator in this domain-specific application,
we seek to enrich the LLM’s domain expertise in this application by
incorporating historical incidents and corresponding root causes as reference
data for the model to perform more reliable and grounded confidence
estimation.

To this end, we present a new framework for confidence estimation. Our method
uses retrieval-augmented large language models (LLMs) to generate calibrated
confidence scores for model-generated root causes and are introduced as LM-
PACE (Prompting and Augmentation for Confidence Estimation with Language
Models). Our proposed method unfolds in two stages. First, to obtain a notion
of the confidence estimator’s uncertainty in examining the predicted root
cause, we prompt the confidence estimator LLM to gauge the strength of
evidence drawn from historical references to assess the current incidents.
Then, we prompt the model to evaluate the candidate root cause generated from
the root-cause predictor based on the references. Finally, we perform an
optimization step to find the most suitable confidence assignment combining
the two scores. Through experimental evaluation, we demonstrate that the model
is able to verbalize its confidence based on the evidence and its own analysis
that can be effectively converted into a well-calibrated score. We answer the
following research questions in the subsequent sections:
RQ1: Can we accurately estimate the confidence for the root cause
recommendations generated by Large language models?
RQ2: Given the importance of domain expertise in cloud incident root cause
analysis, do historical records improve the calibration of the confidence
estimates?
RQ3: Can the model leverage its own analyses to generate more accurately
calibrated scores?
RQ4: Can our proposed approach generalize to encompass different root cause
analysis models?

Through this research, we seek to narrow the gap in the adoption of LLM-based
cloud incident root cause predictors, which aid human engineers by providing
not just predictions but also well-calibrated confidence estimates. Our method
makes minimal assumptions on underlying root cause predictors. We envision our
general framework being adaptable to various AI-assisted IT operations
scenarios, as long as they share similar task configurations and data
availability.

##  2\. Background

###  2.1. Prospects and Challenges of Large Language Models

Pretrained extensively on text corpora from the internet, large language
models (LLMs) have demonstrated impressive capabilities in various tasks
within the NLP domain (Bubeck et al., 2023) and beyond (Deng et al., 2023;
Zhang et al., 2023c). Recent advances in instruction tuning and reinforcement
learning from human feedback of those models are shown to effectively improve
their capabilities in following complex instructions (Ziegler et al., 2019;
Bang et al., 2023; Ouyang et al., 2022).

To adapt LLMs to a specific down-stream task, one can either fine-tune (i.e.
modify the weights of the pre-trained model by performing gradient updates) on
the data of this task or perform in-context learning by showing examples or
verbally explaining the task without updating the model parameters.

Despite being good at language understanding and text generation, the key for
LLMs to successfully answer queries and help humans in various tasks is
knowledge (Izacard et al., 2022). A variety of works have attempted to improve
the performance of LLMs on different tasks (Mialon et al., 2023) by equipping
it with external knowledge. One class of approaches that is particularly
interesting under the context of domain-specific tasks is retrieval
augmentation, where the relevant text chunks are retrieved by a retriever
model based on relevance to the query and presented to the model in the input
(Wang et al., 2023; Guu et al., 2020; Shi et al., 2023). The LLMs, though
lacking that knowledge during the training phase, can comprehend the retrieved
information for its use in generating responses.

While LLMs have exhibited advanced capabilities to produce seemingly coherent
and contextually appropriate responses, they have been observed to frequently
”hallucinate” (Rawte et al., 2023; Zhang et al., 2023a; Ye et al., 2023;
Dhuliawala et al., 2023), producing content that deviates from the original
input, contradicts earlier context, or invents information. These
inconsistencies, well disguised by their eloquence outputs language models,
are subtle and may not be readily discernible to humans. This behavior is
particularly pronounced when the model lacks knowledge upon which to ground
its answers. One prominent example of such a situation would be domain-
specific applications like AIOps.

###  2.2. AI-Assisted Root Cause Analysis and Troubleshooting for Cloud
Incidents

Figure 2. High-level overview of retrieval-augmented confidence calibration.

Cloud incident management is a significant challenge for cloud service
providers, as incidents can result from various issues like code bugs,
dependency failures, and configuration errors. Identifying the root cause is
crucial to resolving the incident. Traditionally, on-call engineers have been
responsible for this task. They analyze incidents by extracting key
information from descriptions and referencing past events and reports.

The rise of LLMs offers an opportunity to loop AI-based systems in root cause
identification. Accurate AI predictions can save engineers considerable time,
eliminating the need to manually review extensive documentation. However, if
AI predictions are incorrect, it can lead to wasted efforts and potential
service disruptions. Engineers face a dilemma: blindly trusting AI can be
risky, but ignoring all the model predictions means missing out on its
benefits.

RCA Model | RCA-Strategy | % Correctness
---|---|---
GPT-4 | Prompting + 8000 Tokens of Most Relevant Historical Incidents | 43.40%
GPT-3.5-Turbo | Prompting + 3896 Tokens of Most Relevant Historical Incidents | 22.30%
GPT-3.5-Turbo | Prompting + 3 Most Relevant Historical Incidents | 23.70%
Text DaVinci-003 | Prompting + 3896 Tokens of Most Relevant Historical Incidents | 30.50%
Text DaVinci-003 | Prompting + 3 Most Relevant Historical Incidents | 25.60%
Table 1. Correctness rates of the root cause predictors on cloud incidents.

As shown in Table 1, though a reasonable percentage of the model-generated
root causes are correct given the difficulty of the task (43.4% for GPT-4;
around or less than 30% for GPT-3.x series), in which cases the
troubleshooting time can be significantly shortened, whereas in the rest of
the cases believing the erroneous model-produced root causes can cause delays
in resolving the issue and restoring services. Another layer of complication
comes from the indiscernible wrong predictions. Figure 4 shows an example of
an incident and the root causes generated by GPT-4 model compared with its
actual root cause as diagnosed by on-call engineers. They are pointing to very
different issues, and following the hint from the model-predicted root cause
would only lead the engineers farther away from mitigating this issue.
However, due to the complex nature of cloud services, engineers would not be
able to tell immediately whether it is correct at first sight based on their
own knowledge - since the model predictions are usually correct-looking.
Therefore they need assistance when deciding whether to follow the model-
generated root cause. Hence, one of the most crucial steps in developing an
AI-assisted root cause analysis system lies in effectively quantifying the
level of trustworthiness associated with each model-generated root cause for
an incident. It ensures the seamless integration of AI technologies in cloud
incident management and fosters improved decision-making and incident
resolution.

###  2.3. Confidence Calibration

Confidence calibration refers to the problem of predicting probability
estimates that represent the true correctness likelihood. Formally, for any
pair of input $X_{i}$ and the model-predicted output $\hat{y}_{i}$ indexed by
$i$ in the dataset, the confidence estimate for the model prediction
$\tilde{P}_{i}(\hat{y}_{i}|X_{i})$ should satisfy

(1) $\mathbbm{P}(l_{i}|\tilde{P}_{i}=p)=p,\forall p\in[0,1]$

, where we use $l_{i}$ to denote the event that $i$-th data point is predicted
correctly. Verbally, the meaning of the equation would be “ the probability of
a data point with confidence $\tilde{P}_{i}=p$ to be correct should be $p$”.
Note that the definition we adopted here is generalized to incorporate
different types of prediction tasks, including open-ended question-answering,
compared to the classic definition for classification where
$\hat{y}_{i},y_{i}\in Y$ are label classes in the label space, and
$l_{i}=\mathbbm{1}(\hat{y}_{i}=y_{i})$. In practice, we often adopt a binning-
based strategy where we divide the probability interval of 0 to 1 into equal
buckets, To estimate the calibrated probability ($p$), we calculate the
proportion of actual positive outcomes that correspond to the predicted
probabilities within each bucket.

To quantify the result of calibration, a commonly used metric is Estimated
Calibration Error (ECE), defined as

(2)
$\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|\text{acc}(B_{m})-\text{conf}(B_{m})\right|$

where $M$ is the total number of bins, $B_{m}$ is the set of indices of
samples that fall into bin $m$, $n$ is the total number of samples,
$\text{acc}(B_{m})$ is the accuracy of bin $m$ calculated as the proportion of
correct predictions in bin $m$, and $\text{conf}(B_{m})$ is the confidence of
bin $m$ calculated as the average predicted probability in bin $m$.

The issue of confidence estimation attains heightened significance in the
context of LLMs due to their direct engagement with human users in multiple
scenarios. As elaborated in Section 2.1, as LLMs are used to perform tasks of
higher complexity, the imperative for assessing the grounding of these models’
predictions in concrete knowledge, as well as the soundness of their reasoning
processes, also increases. This necessitates the development of strategies to
effectively measure the confidence of the model outputs.

Model interpretability gains paramount significance in the symbiosis of human
and machine learning systems. Cosmides and Tooby (Cosmides and Tooby, 1996)
highlighted the innate human ability to comprehend probabilities, underscoring
the value of transparent, interpretable confidence estimates. Such clarity
allows humans to establish an informed level of trust in model outputs, thus
fostering enhanced collaboration with machine learning systems. By integrating
estimated confidence levels with predictions, individuals can adeptly assess
the reliability of the model’s responses, paving the way for informed
decision-making and effective human-machine interaction.

However, confidence calibration in open-ended question-answering is inherently
more challenging than in classification, multiple-choice, or extractive
question-answering. In these other formats, a limited, predefined set of
responses makes confidence calibration easier. However, the infinite
possibilities in open-ended tasks make this process more complex. An
additional layer of difficulty appears when there is potential utilization of
external information since the model is only aggregating information without
possessing it, rendering it incapable of intrinsically evaluating the accuracy
of its responses without careful design.

##  3\. Problem Definition

In this work, we propose to overcome the difficulties in adopting LLM-based
tools to assist humans by confidence estimation. We devise methods to produce
calibrated confidence scores for model-predicted root causes for cloud
incidents to assist on-call engineers in deciding the extent to trust the
model-recommended root cause.

Confidence estimation for recommended root causes presents two critical
challenges that demand careful consideration. The first aspect revolves around
the necessity for confidence estimation methods that are both highly adaptable
and broadly applicable. Given that the settings for root cause predictors -
like the model type used for predictions, the volume of accessible
information, and others - are diverse across services and evolving over time,
the confidence estimation technique should be effective irrespective of the
root cause predictor in place. This adaptability ensures that the confidence
estimation remains reliable across various models and services. Consequently,
it is essential for the confidence estimator to operate independently of the
root cause predictor, ensuring its adaptability and accuracy under varying
circumstances.

Moreover, although language models like GPT-3.5-Turbo and GPT-4 exhibit high
efficiency in diverse applications, essential data, including model parameters
(weights), probabilities, and logits for output tokens, remain unavailable.
These elements are fundamental for confidence calibration. The unavailability
of weights obstructs the possibility of fine-tuning the model to align
confidence, while logits and probability scores are indispensable for
implementing post-hoc calibration techniques. This limitation complicates the
development of dependable confidence estimation procedures and demands an
approach that does not require the above information.

Therefore, a realistic assumption is to treat the root cause generator as a
black box where we only assume access to its generated text, $\hat{r}_{query}$
without knowing anything else, including the configurations, weights, and
output probabilities/logits of the model, the algorithm used to produce the
output, prompt, or any auxiliary information used by the model. Also, the
choice of root cause generator can encompass any model that produces open-
ended textual responses as its predicted root causes. This unrestricted
perspective accounts for the diversity and flexibility of root cause
generation models, accommodating a wide spectrum of approaches. Given the
incident description $d_{query}$ and the predicted root cause
${\hat{r}}_{query}$, our goal is to return a confidence estimate $\psi$ that
reflects the confidence of ${\hat{r}}_{query}$ being the correct root cause.

To this end, our goal is to develop a stand-alone confidence estimator that
remains decoupled from the root cause analysis procedure. By doing so, we can
ensure the versatility and adaptability of our confidence estimation approach,
enabling it to work seamlessly across different language models while
overcoming the limitations of restricted accessibility to certain language
models. In pursuit of this, we need to answer several research questions:
RQ1: Can we accurately estimate the confidence for the root cause
recommendations generated by Large language models?
As previously established, in situations where the root cause generator
operates as a black box, there arises a necessity for an independent
mechanism, incorporating a language model. This module serves the purpose of
comprehending the root causes predicted by the generator and generating
outputs that can be subsequently converted into confidence scores. A
fundamental question is the feasibility of achieving well-calibrated
confidence estimation with a prompting-based approach as well as how to
appropriately design such a framework.

RQ2: Given the importance of domain expertise in cloud incident root cause
analysis, do historical records improve the calibration of the confidence
estimates?
Root cause analysis, together with many other industry application scenarios
of LLMs, poses the same challenge of lacking domain-specific knowledge. In
particular, when requiring the model to analyze the root cause of an incident
critically, it necessarily needs to draw from relevant knowledge - which, due
to the domain-specific nature, is not available from training. We draw
inspiration from the various prior works on in-context retrieval-augmentation
for large language models and see if the model can benefit from relevant
historical incidents to produce more accurate confidence estimates.

RQ3:Can the model leverage its own analyses to generate more accurately
calibrated scores?
Performing a rating/scoring task, especially in such an open-ended situation,
is non-standard and non-trivial for a large language model. Therefore, we
explore whether providing the model with guidance to undertake a preliminary
qualitative evaluation in textual form before proceeding to assign scores can
prove advantageous in this particular context. This approach aims to
investigate the potential efficacy of enhancing the model’s performance by
incorporating an intermediate step of the textual assessment, which could
potentially address the complexities associated with accurate scoring in such
an unconstrained setting.

RQ4:Can our proposed approach generalize to encompass different root cause
analysis models?
As previously highlighted, a pivotal consideration revolves around ensuring
the method’s generalizability to different root-cause distributions from a
variety of root-cause analysis models. To address this concern, we
systematically evaluate the efficacy of the proposed approach across different
sets of root causes generated by various models. By subjecting our method to
this comprehensive testing, we aim to ascertain its robustness and versatility
in accommodating varying characteristics of root-cause predictions.

##  4\. Methodology

Figure 3. An overview of the with LM-PACE calibration method. Note that we are
using different analysis prompts and scoring prompts for COE-scoring and RCE-
scoring respectively.

In our research, we introduce a retrieval-augmented two-step procedure
designed to conduct confidence estimation for model-generated cloud incident
root causes, leveraging the capabilities of large language models through
prompting. The proposed approach involves multiple stages to ensure reliable
and accurate results.

To prepare the retrieval-augmented data, we retrieve a list of similar
historical incidents relevant to each query. This retrieval process is done by
employing semantic similarity-based dense retrievers, which enable us to
identify past incidents with potential connections to the current problem at
hand.

Following the retrieval of historical incidents, we proceed to prompt the
large language model to perform confidence-of-evaluation (COE) pre-
examination. During this stage, the model is given the historical incidents,
and their associated root causes, along with the description of the current
incident. It is then prompted to consider whether it possesses sufficient
information to analyze the underlying cause of the current incident.

.

Example Incident With Model Generating Misleading Root Cause Incident
Description
All machines in SomeService that tries to upgrade OS fails and become
unreachable … we have test VM’s created where the issue reproduces
consistently. The VM comes up healthy. Then after reboot VM goes into the
problematic state. Model-Generated Root Cause
The machines in SomeService faced OS upgrade failures due to incompatibility
with the current machine firmware. When the VMs rebooted post-upgrade, the
outdated firmware caused a conflict, rendering them unreachable. Upgrading the
machine firmware to the latest version is essential to ensure compatibility
and successful OS upgrades. Actual Root Cause The function
KiAdjustGroupConfiguration can cause an uncommitted subnode to be placed in a
committed group. The current KiAllocateProcessorNumber does not account for
this, leading it to recommit the subnode’s group, which reassigns new group
numbers to all subnodes in that group. This disrupts the group database,
causing synchronization issues and potential errors during boot, especially
during the process of ideal node assignment.

Figure 4. An example of a misleading root cause prediction. It is difficult to
decide whether the model-generated root cause is pointing to the correct
troubleshooting direction.

Next, we perform root cause evaluation (RCE) utilizing the information from
the historical incidents. In this step, we ask the language model to evaluate
the generated root cause of the current incident based on the retrieved
historical incidents and their root causes. Finally, to obtain the final
confidence score for each root cause, we combine the COE and RCE scores
together and perform an optimization step. By adopting this retrieval-
augmented procedure, we aim to establish a robust framework for confidence
estimation in cloud incident root causes. This innovative approach empowers us
to leverage the capabilities of large language models effectively, thereby
enhancing our ability to make informed decisions and troubleshoot cloud-
related incidents with increased accuracy and confidence.

###  4.1. Retrieval-Augmented Confidence Calibration

One of the fundamental obstacles in confidence estimation for the root cause
analysis task stems from its strongly domain-specific characteristics. Off-
the-shelf large language models are primarily designed for general-domain
applications and lack the necessary domain-specific knowledge required for a
specific service. On the other hand, relevant historical incidents and the
expert-recommended root causes can offer crucial insights into the current
incidents’ root cause.

Therefore, we implemented a retrieval-based pipeline, utilizing a retriever
module to search for pertinent historical incidents to enhance the confidence
estimator model.

Concretely, we adopted dense retrieval. Given a database
$D=\\{h_{i}=(d_{i},r_{i}):i=1,2,....n{{}_{max}}\\}$ containing historical
incidents $h_{i}$ each of which is a pair of historical incident description
$d_{i}$ and its ground truth root cause $r_{i}$, for each query incident
$d_{query}$, we compute the inner product
$\langle{Enc}(d_{query};\theta_{enc}),{Enc}(d_{i};\theta_{enc})\rangle$ for
each $d_{i}\in D$, where we first compute the embeddings of the descriptions
of query instances and instances in the retrieval database $D$ using the same
encoder LM $Enc$ parametrized by $\theta_{enc}$ and then obtain their
similarity score by calculating the inner product of their encoded vectors.

Then, for each $d_{query}$, we retrieve a sorted list of most relevant
historical incidents from the database based on the similarity scores up to a
fixed token budget $L$ to form the retrieved reference $H$ for the query, that
is, we retrieve the $k$ most relevant examples where $k=max({k^{\prime}})$
$s.t.$ $\mathbf{len}([h_{[1]},...,h_{[k^{\prime}]}])\leq L$ where $h_{[j]}$ is
the $j$-th highest ranked incident with respect to $d_{query}$ and
$\mathbf{len}(.)$ returns the total number of tokens in a list of data
instances. Note that for clarity of notations, we omitted the superscripts
indexing the query incident in this subsection.

Figure 5. Illustration of prompting-based scoring procedure applied in both
COE and RCE steps.

###  4.2. Confidence-of-Evaluation (COE) Pre-examination

The model’s assessment of an incident’s root cause relies significantly on
historical incidents. However, the historical incidents retrieved using the
semantic similarity-based retriever might encompass a range of situations,
differing in relevance to the current incident. Consequently, these incidents
might not offer sufficient guidance to the model for precisely evaluating the
root cause of the given query incident. Additionally, given the recognized
occurrence of hallucination in large language models as discussed in Section
2.1, there is a varying degree of reliability in the model’s judgments.
Therefore, this step aims to address the varying levels of hallucination and
assess the extent to which the model’s evaluation of the root cause is based
on factual and reliable information.

The confidence-of-evaluation (COE) estimator takes in the retrieved historical
incidents $H=\\{h_{1},...h_{k}\\}$ ( where each $h_{i}=(d_{i},r_{i})$ is a
pair of historical incident descriptions and its ground truth root cause)
together with $d_{query}$ which is the query incident, and decides whether we
have enough evidence from $H$ to reason about the root cause of the current
incident. The goal of this step is to obtain the model’s level of confidence
in its ability to reason effectively about the root cause of the query
incident given the retrieved information. If the model is low in confidence of
determining the root cause due to the lack of information, its evaluation of
the generated root cause would also be less trustworthy.

As observed in works studying problem-solving with LLMs (Nye et al., 2021;
Dhuliawala et al., 2023) the LLMs benefit from simulating the process of
articulating their intermediate cognitive processes while addressing a task.
Through the verbalization of thought processes, the model gains the ability to
meticulously plan its approach to a question, elucidate intricate steps of
reasoning, deconstruct complex problems into manageable sub-problems, and
retain and utilize partial deduction outcomes and interim computation results.
Ultimately, this method empowers the model to arrive at a well-considered and
logical final decision.

In our scenario, given that both deciding on the retrieved incidents’
suggestive power and quantifying the confidence of model-predicted root causes
are inherently complicated and multifaceted in nature, we seek to elicit the
model’s ability to make judgments by inducing it to produce intermediate
analysis.

Concretely, as demonstrated in Figure 5, the model first needs to generate
analysis in textual form conditioning on the historical incidents, description
of the current incident, and the instruction for analysis $I{{}_{a}^{COE}}$.
In order to reduce biases and cover more facets, we sample multiple analyses
for each query (Li et al., 2023). $a{{}_{ij}^{c}}\sim
p(a|H,d_{query},I{{}_{a}^{COE}})$ where $j=1,2,...,k_{1}$ and $k_{1}$ is the
number of analyses for each incident.

Then we sample $k_{2}$ binary responses from model on whether the model thinks
the historical conditioning on each analysis, that is, for each
$a{{}_{i}^{c}}$, we sample responses $c{{}_{j}^{i}}\sim
p(c|H,d_{query},I{{}_{s}^{COE}},a{{}_{i}^{c}})$ and estimate the confidence-
of-evaluation (COE) score with the empirical mean across all scores obtained
from all analyses

$E(c)=\frac{1}{k_{1}\times
k_{2}}\sum_{j=1,...,{k_{2}}}\sum_{k=1,...,{k_{1}}}c{{}_{j}^{i}}$

. In this step, we provide the model with a multiple-choice question
consisting of two options (Yes/No) and ask it to pick the letter of choice and
$c{{}_{j}^{i}}=\mathbbm{1}(\text{choice}==\textbf{yes})\in\\{0,1\\}$.

###  4.3. Root Cause Evaluation (RCE) Scoring

In the second step of the confidence estimation, we ask the model to evaluate
the candidate’s root cause based on the historical incidents. The root cause
evaluator first takes in the same $H=\\{h_{1},...h_{k}\\}$ again as in COE
step, together with $(d_{query},{\hat{r}}_{query})$ where ${\hat{r}}_{query}$
is the predicted root cause to be evaluated, to analyze whether the predicted
root cause ${\hat{r}}_{query}$ is a plausible root cause to the query incident
based on historical records. Again, we first ask the model to provide with
analysis $a{{}_{ij}^{s}}$’s following certain rubrics we specified in the
instruction $I{{}_{a}^{RCE}}$, and then ask the model to score following the
instruction $I{{}_{s}^{RCE}}$.

Similar to the COE step, we sample ${k_{1}}^{\prime}$ analyses for each
incident. Yet, we take on a different scoring mechanism. The evaluation of
root cause is more complicated as there are multiple dimensions including
truthfulness (i.e. whether it contains false information), groundedness (i.e.
to what extent the historical incidents support or go against the generated
root cause), and informativeness (i.e. the level of detail in the generated
root cause, and its adequacy in directing engineers for troubleshooting,
relative to the guidance offered for analogous historical incidents).
Therefore, instead of simply asking for binary responses, we prompt the model
to produce scores on a specified scale conditioning on each analysis. The
score is generated from $s{{}_{j}^{i}}\sim
p(s|H,d_{query},{\hat{r}}_{query},I{{}_{s}^{RCE}},a{{}_{i}^{s}})$. Similar to
step-1, we again aggregate the mean score

$E(s)=\frac{1}{{k_{1}}^{\prime}\times{k_{2}}^{\prime}}\sum_{j=1,...,{k_{2}}^{\prime}}\sum_{k=1,...,{k_{1}}^{\prime}}s{{}_{j}^{i}}$

where we again ask the model to produce ${k_{2}}^{\prime}$ scores conditioning
on each analysis.

###  4.4. Estimating Confidence From COE and RCE Scores

Given the COE and RCE scores obtained from two-step confidence estimation for
each data point, the problem remains to be finding an optimal mapping from
these intermediate scores into confidence.

Consider a problem where we assumed $m$ categories of confidence level evenly
dividing the interval $[0,1]$, and that want to assign each root cause the
most appropriate category that best indicates the confidence level. Let
$\pi(.,.)$ denote the score transformation function that takes in COE and RCE
scores and maps them into a value between 0 and 1, and $\omega(.)$ be the
weighting function that determines the relative importance of each category,
$\mathbbm{1}[l_{j}]$ denote the event that the generated root cause $j$ is
correct, and $t_{0},\ldots,t_{m}$ be thresholds for each category where
$t_{0}=0$ and $t_{m}=1$ are minimum and maximum possible scores from
$\pi(.,.)$, respectively.

In general, optimization objective takes the form

(3)
$\underset{t_{0}<t_{1}<\ldots<t_{m},\theta}{\text{minimize}}\sum_{i=1}^{m-1}\omega(i)\left|\pi({c_{j},s_{j}})-\frac{\sum_{j}\mathbbm{1}[t_{i}\leq\pi({c_{j},s_{j}})\leq
t_{i+1}]*\mathbbm{1}[l_{j}]}{\sum_{j}\mathbbm{1}[t_{i}\leq\pi({c_{j},s_{j}})\leq
t_{i+1}]}\right|$

The parameter $\omega$ plays a pivotal role in adjusting the relative
significance attributed to each confidence bucket. For example, in situations
where emphasizing high-confidence cases is the top priority, we have the
flexibility to allocate greater weightage to the respective bins. In case of
$\omega(i)=\sum_{j}\mathbbm{1}[t_{i}\leq\pi({c_{j},s_{j}})\leq t_{i+1}]$, the
objective reduces to Expected Calibration Error (ECE) score. In our
experiment, we take ECE objective, while the weighting mechanism of each bin
can be tailored for different scenarios.

##  5\. Experiments

###  5.1. Experiment Set-up

####  5.1.1. Dataset and Labels

In this subsection, we present our dataset preparation process.

Our collection of data consists of incident data gathered from different
services and levels of severity within CompanyX, amounting to a total of
121,308 incidents. In order to create this dataset, we chose 98,308 incidents
as the retrieval set, 2,000 for the validation set, and the remaining 3,000
for the testing set. We search for the best set of parameters that optimizes
the objective (ECE) on the validation set and subsequently perform evaluation
on the test set.

For each validation and test instance, we set the maximum token budget $L$ for
retrieved incidents to 3896 and retrieve the most similar historical incidents
within the budget. The average number of historical incidents is around 15.

Due to the cost of human labeling, we generated pseudo-labels with GPT-4 by
feeding it with pairs of $(r_{i},\hat{r_{i}})$ and ask it whether these are
similar. We ask the model to provide with ratings in integer values from 1 to
3. We sample 128 scores per query and query 4 times per input. We compute the
average score across the 512 outputs. We decided the threshold for correctness
to be $\geq 2.3$ which gives the maximum $F1$ score on an independent
validation set with human annotations.

####  5.1.2. Evaluation Protocol

We demonstrate the effectiveness of the method via reliability diagrams, ECE
scores, and human evaluation. Reliability diagrams (6 and 12) serve as vital
tools for evaluating the calibration of probabilistic predictions in machine
learning models. A reliability diagram visually compares the predicted
probabilities produced by a model against their observed frequencies, offering
insights into the model’s ability to provide well-calibrated confidence
levels. It serves as a rapid assessment of the performance of calibration. In
the context of the reliability diagram, the horizontal axis is the confidence
axis, representing a range between 0 and 1. This interval is equally divided
into smaller sub-intervals. The height of bars in each sub-interval indicates
the proportion of positive labels among the collection of root causes that
possess an assigned confidence score lying within each respective sub-
interval. The regions shaded in gray are intervals within which the
corresponding bars are expected to align.

In addition, we also presented the Expected Calibration Error (ECE) score
which stands as a pivotal metric in quantifying the calibration quality of
probabilistic predictions generated by machine learning models. ECE gauges the
disparity between predicted probabilities and observed frequencies across
various confidence intervals. This assessment offers a nuanced perspective on
the model’s calibration precision, highlighting potential deviations from
ideal alignment. By numerically quantifying these discrepancies, ECE
facilitates an objective evaluation of a model’s reliability in terms of its
probability estimates, aiding practitioners in pinpointing and addressing
areas requiring calibration enhancement.

####  5.1.3. Root Cause Generators

To assess the efficacy of the proposed approach across diverse black-box
models, we conducted experiments involving three distinct language models as
root-cause generators that include models fine-tuned both with (GPT-4,
GPT-3.5-Turbo) and without human feedback (Text-DaVinci-003) which are
expected to produce different output distributions in terms of style, tone,
and correctness. These experiments were conducted under various configurations
as discussed below.

We explored models exposed to varying degrees of historical information. The
annotation ”-3-shot” denotes root causes generated based on the three most
similar historical instances based on semantic similarity of incident
descriptions, while the annotation ”-3896” corresponds to root causes derived
from the maximum feasible number of the most similar historical incidents
while adhering to a constraint of 3896 tokens for the prompt size. We
established a maximum limit of 200 tokens for the root cause prediction. ”-8K”
denotes 8000 tokens of most similar historical incidents.

####  5.1.4. Implementation Details

For conducting confidence estimation in our experiment, we used OpenAI GPT-4
(8K maximum tokens) as our base language model. This powerful model serves as
the foundation for assessing the confidence levels associated with each model-
generated root cause. During the process of generating analyses and scoring,
we set the temperature parameter ($T$) to 1. The temperature value allows us
to control the level of randomness in the model’s output. When generating
analyses during confidence estimation, we limit the maximum completion token
to 256.

###  5.2. Experimental Results

####  5.2.1. RQ1: Can we accurately estimate the confidence for the root cause
recommendations generated by Large language models?

Figures 6(b) 12(a) 12(b) 12(c) and 12(d) are reliability diagrams plotted for
the confidence estimates by our full method for root causes generated by
various models under different conditions. As demonstrated in the reliability
diagrams, our proposed approach is able to produce well-calibrated confidence
estimates for the root causes generated under different root cause generator
set-ups.

To assess the significance of the COE score, we conducted the same
optimization procedure solely on the basis of the RCE score (see Figures 12(e)
12(f) 12(g) 12(h)). Notably, the confidence scores are seriously mis-
calibrated when using only RCE scores. In the confidence estimation procedure,
the COE score plays an important role in producing a calibrated confidence
score since it counters the score hallucination effect in situations where the
calibrator model is not well-informed enough to present a fair evaluation of
the root cause. In Figure 8, we juxtapose the distributions of RCE scores with
calibrated scores for incorrect root causes. The evident disparity between
these two score distributions underscores that relying solely on the RCE score
is inadequate for generating well-calibrated confidence estimates.

We analyze the distributions of cases in which the model assigns high COE
scores, indicating instances where it deems itself equipped with ample
evidence to engage in reasoning about the underlying factors. This is
contrasted with cases receiving low scores. In situations of low COE scores,
the model demonstrates a tendency to overestimate incorrect root causes (see
Figure 10(b)), while undervaluing correct root causes (See Figure9(a)). This
divergence is less pronounced in instances with high COE scores. It implies
that the model is indeed more confused when there is less evidence from
historical incidents. This observation underscores the vital importance of
considering the model’s confidence in estimation and integrating such insights
into the confidence estimation framework.

Figure 6(c) shows the reliability diagram if we were to project scores
linearly to the interval between 0 and 1 and Figure 6(d) is the linearly
projected RCE score. By comparing Figures 6(b) and 6(c), we show that the
optimization step for binning is necessary since neither model-produced score
is scaled uniformly.

####  5.2.2. RQ2: Given the importance of domain expertise in cloud incident
root cause analysis, do historical records improve the calibration of the
confidence estimates?

(a) Confidence Estimation by full LM-PACE

(b) Confidence Estimation Without COE estimation.

(c) Confidence Estimation by Uniform Binning Strategy.

(d) Confidence Estimation by Ignoring COE and Uniform Binning.

(e) Confidence Estimation without Context and Calibrated Binning Strategy.

(f) Confidence Estimation without Analysis and Calibrated Binning Strategy.

(g) Confidence Estimation without Analysis & without Context and Calibrated
Binning Strategy.

Figure 6. Reliability curve of calibrated confidence on root causes generated
by GPT-4 with 8K tokens of context (including query incident).

Our empirical study has provided compelling evidence of the significant
advantages gained from incorporating historical incidents into the RCE scoring
process. Without the introduction of this additional information, the RCE
procedure would have lost its essential purpose and efficacy. Due to the
highly domain-specific nature of the problem, it inherently relies on external
sources of information that were not present in its original pre-training
corpus. This stands in contrast to tasks involving commonsense question-
answering or general reasoning in the NLP domain, where the necessary
knowledge is already available to the model via pre-training.

In scenarios where the model must generate analyses and judgments about root
causes without the context of historical incidents, it tends to lean on
general-domain knowledge and superficial correspondences between incident
descriptions and potential root causes. Such reliance is often insufficient,
because the model, without historical context or service-specific insights,
tends to fail to grasp the nuanced, context-specific elements and foundational
patterns essential in thoroughly examining predicted root causes. Incident
descriptions alone do not encapsulate the root cause, leading to a lack of
depth and precision in the model’s analysis. Consequently, in the absence of
historical incidents and service-specific knowledge, the accurate evaluation
of a root cause proves impossible.

By contrast, in scenarios where historical incidents are provided, the model
becomes capable of analyzing the similarities between the current incident and
past occurrences, allowing it to compare the model-generated root causes with
the actual root causes of these historical incidents.

A High-Confidence Prediction Model Generated Root Cause Physical ARM64
machines experience occasional failures due to lost heartbeat or bad updates,
requiring intervention to recover. Transitioning to ARM64 VMs could help
reduce downtime and improve the pool’s capacity. Ground Truth Root Cause
Physical ARM64 machines sometimes lose heartbeat on reboot or are affected by
a bad update and require intervention to recover. If possible, transition to
ARM64 VMs could reduce our time spent recovering physical machines. Figure 7.
An Example of model-generated root cause with high confidence. The model
predicted root cause indeed is correct.

The integration of historical data not only enhances its understanding of
complex, domain-specific nuances. With access to a rich database of past
incidents, the model becomes adept at recognizing patterns and identifying
recurring issues. In cases where a recurring issue is detected, the confidence
estimator is empowered to directly verify the accuracy of the identified root
cause.

(a) GPT-4-8K.

(b) Text-DaVinci-003-3shot.

(c) GPT-3.5-Turbo-3shot.

Figure 8. Frequency histogram of scores for incorrect root cause predictions.
Distributions of RCE scores and full confidence score are very different.

####  5.2.3. RQ3: Can the model leverage its own analyses to generate more
accurately calibrated scores?

The emerging evidence surrounding chain-of-thought reasoning presents a
promising avenue for bolstering the capabilities of a various spectrum of
natural language processing (NLP) reasoning tasks. In our pursuit of
harnessing this potential, we set our sights on investigating whether this
approach could be seamlessly integrated into the scoring process, which is
inherently characterized by subjectivity and open-endedness. Upon conducting
our investigations, we observed that the language model does, in fact, derive
substantial benefits from its earlier analyses which serves as a well-informed
basis for it to make more reasonable and accurate scoring.

Notably, for COE scoring, direct scoring yields low scores for the majority of
the cases since the question is highly non-trivial for the model to answer
directly, thus the model tends to be confounded and biased towards generating
low scores regardless of how informed it actually is, as shown in Figure 10.
For RCE, zero-shot scoring tends to perform worse in separating the correct
and incorrect cases (Figure 9), thus yielding unsatisfactory confidence
estimation results.

The empirical results imply that generating analysis is an important component
of our approach in this task, as it empowers the model to provide
comprehensive insights while addressing the specific questions outlined in the
instructions. By generating analysis, the language model can delve deeper into
the given context, extracting relevant information, and offering well-
structured responses that cater precisely to the task at hand. Through this
process, the model can identify key examples within the retrieved context that
hold particular significance in making informed judgments. By highlighting
these crucial examples, the model enhances its ability to prioritize relevant
information for the RCE step, ultimately leading to more accurate responses.

(a) High ground-truth score data points.

(b) Low ground-truth score data points.

Figure 9. Distribution of RCE scores.

(a) Distribution of COE scores with/without analysis.

(b) Distribution COE score difference w./w.o. analysis for each incident.

Figure 10. COE scores. Lack of analysis makes the model unable to leverage its
analysis to score rationally and therefore yields meaninglessly low scores.

####  5.2.4. RQ4: Can our proposed approach generalize to encompass different
root cause analysis models?

Root Cause Generator | LM-PACE |  RCE-Only With Optimization |  COE and RCE Uniform Binning |  RCE-Only Uniform Binning
---|---|---|---|---
GPT-4-8K | 0.082 |  0.157 (+0.075) |  0.118 (+0.036) |  0.261 (+0.179)
GPT-3.5-Turbo-3shot | 0.084 |  0.192 (+0.108) |  0.121 (+0.037) |  0.327 (+0.243)
Text-DaVinci-003-3shot | 0.081 |  0.219 (+0.138) |  0.132 (+0.051) |  0.459 (+0.378)
GPT-3.5-Turbo-3896 | 0.053 |  0.156 (+0.103) |  0.112 (+0.059) |  0.355 (+0.302)
Text-DaVinci-003-3896 | 0.072 |  0.189 (+0.117) |  0.176 (+0.104) |  0.395 (+0.323)
Table 2. ECE scores under different settings. Numbers in parentheses are ECE
differences compared with the full method. Lower ECE scores indicate better
calibration performance. Full method always perform the best in this table.

Here we examine whether the proposed method can indeed work for root causes
generated by different upstream root cause generators with various amounts of
retrieved information, which implies varying accuracies for upstream root
cause analysis results as well as the distributions of the underlying text.

(a) Human evaluation procedure with consensus labeling.

(b) Human-evaluation results for high-confidence cases.

Figure 11. Human Evaluation

By conducting experiments using the underlying causes produced by retrieval-
augmented Text-DaVinci-003 applied to the three most pertinent incidents
(depicted in Figure 12(a)), along with GPT-3.5-Turbo’s responses to the top
three relevant incidents (illustrated in Figure 12(b)), as well as in
scenarios with maximum context length (as presented in Figure 12(c)), we
demonstrate the persistent effectiveness of the proposed approach. This
applicability holds true across diverse output distributions from models both
fine-tuned with human feedback (such as GPT-3.5-Turbo and GPT-4) and those
that are not (like Text-DaVinci-003), as well as when exposed to varying
amounts of information.

(a) Calibration Result for Text-DaVinci-003-3shot.

(b) Calibration Result for GPT-3.5 Turbo-3shot.

(c) Calibration Result for GPT-3.5-Turbo-3896.

(d) Calibration Result for Text-DaVinci-003-3896.

(e) Calibration Result for Text-DaVinci-003-3shot w/o COE.

(f) Calibration Result for GPT-3.5-Turbo-3896 w/o COE.

(g) Calibration Result for GPT-3.5-Turbo-3896 w/o COE.

(h) Calibration Result for Text-DaVinci-003-3896 w/o COE.

Figure 12. Reliability Diagrams of GPT-3.5-Turbo and Text-DaVinci-003
Experiments. Our approach produces accurately calibrated scores in these
settings.

###  5.3. Human Evaluation

The human evaluation setup involves assessing the quality of model-generated
root causes in comparison to the ground truth root causes. This evaluation
process is performed by human evaluators who are tasked with assigning a
similarity score to each model-generated root cause based on a predefined
scale. The goal is to determine how well the model’s outputs align with the
actual root causes. We present 5 options to human labelers (including the
score and the corresponding tier):
1 (Totally Irrelevant); 2 (Vague or Distant Relevance); 3 (Has Similarities);
4 (Clearly Similar); 5 (Very Similar and Highly Relevant).
We employ a consensus labeling approach, wherein an item receives a label if a
consensus is achieved among the minimum required number of labelers. Should a
consensus not be reached, the item undergoes further assessment by additional
labelers. The minimum required number of labelers in our case is 3.

In real-world cloud incident troubleshooting scenarios, the most critical
instances are those where the model assigns high confidence scores to the
predicted root causes. In such cases, on-call engineers are more inclined to
trust these predictions, leading to significant consequences if the estimates
prove inaccurate.

Therefore, we labeled the root causes generated from GPT-4 8K token falling
into the 80% - 100% bin in the full method, the label distribution is shown in
Figure 11(b). We notice that 86% of the cases (scored 5 and 4) are labeled by
humans as highly similar to ground truth root causes, which falls within the
range of 80% - 100%. The human evaluation results further demonstrate that the
calibration approach is indeed effective as it can produce results consistent
with human judgment.

##  6\. Related Work

###  6.1. Automated Root Cause Analysis with Language Models

Machine learning as applied to IT operations, often termed as MLOps or AIOps,
has been studied in the context of cloud incident management for a while. For
close to a decade, various studies have underscored its effectiveness in
managing and resolving incidents in cloud services(Chen et al., 2019a, b; Azad
et al., 2021; Nair et al., 2015; Bansal et al., 2020; Luo et al., 2014; Jiang
et al., 2020). The progression in this field has seen a myriad of approaches;
from methodologies that leverage time-series and feature-based techniques to
the ones employing advanced generative language models. The underlying motive
has always been to harness the power of the colossal amount of data available,
conceptualize effective machine-learning solutions, and thereby, augment human
capabilities and enhance overall productivity.

In the recent past, we’ve witnessed a paradigm shift, particularly in the
capabilities of language models. These models have not only evolved but also
showcased proficiency in managing domain-specific question-answering tasks
(Wang et al., 2023). The implications of such advancements are profound. They
have carved out potential pathways where these models can be seamlessly
integrated into the product’s life cycle. A case in point is the root cause
analysis (RCA) procedure which has also been studied by a series of works like
(Ahmed et al., 2023; Chen et al., 2023; Jin et al., 2023). However, one common
problem is that language models are knowingly good at hallucinating (Zhang et
al., 2023b; Ji et al., 2023), which can generate outputs that are seemingly
plausible while containing highly misleading information that could confuse
the stakeholders taking the model outputs as reference.

###  6.2. Confidence Calibration Of Pre-trained Language Models

Confidence calibration for pre-trained language models has been studied under
several question-answering set ups. They explored various methods to calibrate
pre-trained language models for QA tasks. Mielke et al. (Mielke et al., 2022)
proposed to train a separate confidence calibrator, but their approach
requires representations from the models. Lin et al (Lin et al., 2022)
proposed to use the verbalized confidences by the models as confidence
estimates, but it requires fine-tuning the model that is used to produce
answers to perform this function, which is not always feasible.

Confidence estimation under an in-context learning set-up is more challenging
as it is more constrained in terms of the components we can manipulate. It has
been pointed out by Zhou et al. (Zhou et al., 2023) that guiding models to
express their uncertainties verbally via prompting could harm the model’s
performance. Zhao et al. (Zhao et al., 2023) proposed a calibration framework
tailored for classification tasks, requiring finite output space and a hand-
crafted supervision function, and therefore cannot be applied to open-ended
question-answering problems. Beyond the categorical set-up, recent works
(Xiong et al., 2023; Tian et al., 2023) explored prompting-based confidence
estimation for factual and numerical QA tasks. Xiong et al. (Xiong et al.,
2023) pointed out that their confidence estimation approaches struggle in more
complex tasks particularly those requiring domain-specific knowledge. In
particular, Jiang et al. (Jiang et al., 2021) demonstrated that augmenting
inputs with retrieved knowledge can benefit confidence estimation in question-
answering. However, their study was conducted over either multiple-choice or
extractive QA tasks, both of which are essentially classification problems
without needing to generate open-ended responses.

##  7\. Conclusion

In light of our exploration into the challenges of the rapidly evolving AIOps
applications, particularly concerning cloud-based platforms, this paper
identifies the major obstacles tied to the adoption of LLM-based root cause
predictors. These include the potential for incorrect recommendations and
undetectable hallucinations, issues that could misdirect on-call engineers and
lead to substantial losses. We propose to perform confidence estimation for
each prediction in order to mitigate these challenges. We developed a general
framework that makes minimal assumptions on the root cause predictor, LM-PACE,
designed with the flexibility to encompass various root cause predictors. Our
approach breaks the confidence estimation process into two phases, where we
first measure the strength of evidence provided by historical references, then
quantify the quality of the model prediction. We obtain final confidence
assignments from the two scores with an optimization step. We have
demonstrated the adaptability and effectiveness of our approach across diverse
output distributions. Notably, this research stands as a pioneering effort in
prompting retrieval-augmented LLMs for confidence calibration in open-ended
question-answering tasks in domain-specific scenarios. More broadly, this work
also serves as evidence to prove the ability of LLMs to be prompted to produce
calibrated confidence estimates. It also suggests their potential to be used
for confidence estimation in various other domain-specific tasks to foster
more reliable AIOps systems.

###### Acknowledgements.

We thank Yangyi Chen, Chi Han, Jialiang Xu, and Liliang Ren for their
constructive discussions and valuable feedback on this work.

## References

  * (1)
  * Ahmed et al. (2023) Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. 2023.  Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models.  arXiv:2301.03797 [cs.SE]
  * Azad et al. (2021) Amar Prakash Azad, Supriyo Ghosh, Ajay Gupta, Harshit Kumar, and Prateeti Mohapatra. 2021.  Picking Pearl From Seabed: Extracting Artefacts from Noisy Issue Triaging Collaborative Conversations for Hybrid Cloud Services.  arXiv:2105.15065 [cs.AI]
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023\.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _arXiv preprint arXiv:2302.04023_ (2023).
  * Bansal et al. (2020) Chetan Bansal, Sundararajan Renganathan, Ashima Asudani, Olivier Midy, and Mathru Janakiraman. 2020.  DeCaf. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice_. ACM.  https://doi.org/10.1145/3377813.3381353
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  arXiv:2303.12712 [cs.CL]
  * Chen et al. (2019a) Junjie Chen, Xiaoting He, Qingwei Lin, Yong Xu, Hongyu Zhang, Dan Hao, Feng Gao, Zhangwei Xu, Yingnong Dang, and Dongmei Zhang. 2019a.  An empirical investigation of incident triage for online service systems. In _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 111–120.
  * Chen et al. (2019b) Junjie Chen, Xiaoting He, Qingwei Lin, Hongyu Zhang, Dan Hao, Feng Gao, Zhangwei Xu, Yingnong Dang, and Dongmei Zhang. 2019b.  Continuous Incident Triage for Large-Scale Online Service Systems. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 364–375.  https://doi.org/10.1109/ASE.2019.00042
  * Chen et al. (2023) Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, Jun Zeng, Supriyo Ghosh, Xuchao Zhang, Chaoyun Zhang, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023.  Empowering Practical Root Cause Analysis by Large Language Models for Cloud Incidents.  arXiv:2305.15778 [cs.SE]
  * Cosmides and Tooby (1996) Leda Cosmides and John Tooby. 1996.  Are Humans Good Intuitive Statisticians After All? Rethinking Some Conclusions from the Literature on Judgment Under Uncertainty.  _Cognition_ 58 (01 1996), 1–73.  https://doi.org/10.1016/0010-0277(95)00664-8
  * Deng et al. (2023) Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023.  Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt.  _arXiv preprint arXiv:2304.02014_ (2023).
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.  Chain-of-Verification Reduces Hallucination in Large Language Models.  arXiv:2309.11495 [cs.CL]
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training. In _International conference on machine learning_. PMLR, 3929–3938.
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_ (2022).
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of Hallucination in Natural Language Generation.  _ACM Comput. Surv._ 55, 12, Article 248 (mar 2023), 38 pages.  https://doi.org/10.1145/3571730
  * Jiang et al. (2020) Jiajun Jiang, Weihai Lu, Junjie Chen, Qingwei Lin, Pu Zhao, Yu Kang, Hongyu Zhang, Yingfei Xiong, Feng Gao, Zhangwei Xu, Yingnong Dang, and Dongmei Zhang. 2020.  How to Mitigate the Incident? An Effective Troubleshooting Guide Recommendation Technique for Online Service Systems. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 1410–1420.  https://doi.org/10.1145/3368089.3417054
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering.  _Transactions of the Association for Computational Linguistics (TACL)_ 9 (9 2021), 962–977.
  * Jin et al. (2023) Pengxiang Jin, Shenglin Zhang, Minghua Ma, Haozhe Li, Yu Kang, Liqun Li, Yudong Liu, Bo Qiao, Chaoyun Zhang, Pu Zhao, Shilin He, Federica Sarro, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023.  Assess and Summarize: Improve Outage Understanding with Large Language Models.  arXiv:2305.18084 [cs.SE]
  * Li et al. (2023) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023.  Making Language Models Better Reasoners with Step-Aware Verifier. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Toronto, Canada, 5315–5333.  https://doi.org/10.18653/v1/2023.acl-long.291
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Teaching Models to Express Their Uncertainty in Words.  arXiv:2205.14334 [cs.CL]
  * Luo et al. (2014) Chen Luo, Jian-Guang Lou, Qingwei Lin, Qiang Fu, Rui Ding, Dongmei Zhang, and Zhe Wang. 2014.  Correlating Events with Time Series for Incident Diagnosis. In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (New York, New York, USA) _(KDD ’14)_. Association for Computing Machinery, New York, NY, USA, 1583–1592.  https://doi.org/10.1145/2623330.2623374
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023.  Augmented Language Models: a Survey.  _Transactions on Machine Learning Research_ (2023).  https://openreview.net/forum?id=jh7wH2AzKK Survey Certification.
  * Mielke et al. (2022) Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.  Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 857–872.  https://doi.org/10.1162/tacl_a_00494
  * Nair et al. (2015) Vinod Nair, Ameya Raul, Shwetabh Khanduja, Vikas Bahirwani, Qihong Shao, Sundararajan Sellamanickam, Sathiya Keerthi, Steve Herbert, and Sudheer Dhulipalla. 2015.  Learning a hierarchical monitoring system for detecting and diagnosing service issues. In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_. 2029–2038.
  * Nye et al. (2021) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021\.  Show your work: Scratchpads for intermediate computation with language models.  _arXiv preprint arXiv:2112.00114_ (2021).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023.  A Survey of Hallucination in Large Foundation Models.  _arXiv preprint arXiv:2309.05922_ (2023).
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_ (2023).
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023.  Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.  arXiv:2305.14975 [cs.CL]
  * Wang et al. (2023) Zezhong Wang, Fangkai Yang, Pu Zhao, Lu Wang, Jue Zhang, Mohit Garg, Qingwei Lin, and Dongmei Zhang. 2023.  Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering.  arXiv:2305.11541 [cs.CL]
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023.  Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.  _arXiv preprint arXiv:2306.13063_ (2023).
  * Ye et al. (2023) Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2023.  Cognitive Mirage: A Review of Hallucinations in Large Language Models.  arXiv:2309.06794 [cs.CL]
  * Zhang et al. (2023b) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2023b.  How Language Model Hallucinations Can Snowball.  arXiv:2305.13534 [cs.CL]
  * Zhang et al. (2023c) Shizhuo Dylan Zhang, Talia Ringer, and Emily First. 2023c.  Getting More out of Large Language Models for Proofs.  arXiv:2305.04369 [cs.FL]
  * Zhang et al. (2023a) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023a.  Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.  arXiv:2309.01219 [cs.CL]
  * Zhao et al. (2023) Theodore Zhao, Mu Wei, J Samuel Preston, and Hoifung Poon. 2023.  Automatic Calibration and Error Correction for Large Language Models via Pareto Optimal Self-Supervision.  _arXiv preprint arXiv:2306.16564_ (2023).
  * Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023.  Navigating the Grey Area: Expressions of Overconfidence and Uncertainty in Language Models.  arXiv:2302.13439 [cs.CL]
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-Tuning Language Models from Human Preferences.  _ArXiv_ abs/1909.08593 (2019).  https://api.semanticscholar.org/CorpusID:202660943


NOTES ON THE INVERSE COMPTON SCATTERING

K.A. Bornikov1, I.P. Volobuev2, Yu.V. Popov2,3

1Physical Faculty, Lomonosov Moscow State University, Moscow, Russia

2Skobeltsyn Institute of Nuclear Physics, Lomonosov Moscow State University,
Moscow, Russia

3Bogoliubov Laboratory of Theoretical Physics, Joint Institute for Nuclear
Research,

Dubna, Moscow Region, Russia

###### Abstract

The paper deals with kinematic conditions for the inverse Compton scattering
of photons by relativistic electrons and the polarizations of the colliding
particles, which affect the value of the differential cross section of the
process. A significant influence of the electron and photon helicity on the
value of the cross section has been found. In the ultrarelativistic case, a
surprising effect of an almost twofold increase in the cross section of
scattering in the direction of the initial electron momentum has also been
discovered, when the initial photon momentum is transverse to that of the
initial electron.

## INTRODUCTION

At the present time, the state pays a great attention to the development of
neutron and synchrotron research. ”Federal scientific and technical program
for the development of synchrotron and neutron research and research
infrastructure for 2019 –2027 years” [1] has been announced. In the context of
our work, we would like to highlight the following topics of this program:
”Methods of synchrotron and neutron diagnostics of materials and nanoscale
structures for promising technologies and technical systems, including
fundamentally new nature-like component base,” ”Methods of synchrotron and
neutron studies of structure and dynamics of biological systems at different
levels of organization (biomolecules, macromolecular complexes, viruses,
cells),” ”New technologies of electron and proton accelerators necessary for
development of new sources of synchrotron radiation of the 4th and subsequent
generations, X-ray laser free electrons and pulsed neutron sources”, etc. This
program is directly related to the inverse Compton scattering.

The effect discovered by Compton 100 years ago [2] consisted in a decrease in
the frequency (energy) of a photon as a result of its scattering by a charged
particle (electron) at rest. This discovery was of an enormous methodological
importance for quantum mechanics, since it confirmed the behavior of the
photon as a particle and formed the basic concept of wave-particle duality in
quantum mechanics. From the very beginning the Compton effect was aimed at
studying the distribution of the electron momentum in the targets, which
originally were predominantly solid-state ones [3]. Recently, it was found
that experiments on Compton scattering of photons at bound electrons could be
conducted with gas targets and slow (cold) atoms using the COLTRIMS detector
[4]. The theory of precisely such nonrelativistic experiments can be found, in
particular, in papers [5, 6]

Much later, the attention of scientists, and especially engineers, was
attracted by the so-called inverse Compton effect, which gave a possibility to
build relatively compact X-ray and even gamma radiation sources [7, 8]. The
inverse Compton effect consists in the (head-on) collision of a photon beam
with a beam of relativistic charged particles (usually electrons). The
scattered photons acquire much more energy than the initial photons, and they
can be used further, which is largely an engineering challenge. Some aspects
of Compton scattering at nucleons are presented in [9, 10], although, as shown
below, protons are hardly appropriate to be used for the purposes of the
inverse Compton effect in the laboratory. The inverse Compton effect often
manifests itself in astrophysical processes involving ultrarelativistic
particles [11, 12, 13, 14, 15].

The final photons collected and collimated into a beam can be used to
investigate deep levels of heavy atoms and atomic nuclei. In this case, it is
possible to study the Compton excitation of nuclei, various photonuclear and
photoatomic processes [16], and even to use high-energy photons in such
fantastic projects as tomography of ultrarelativistic nuclei by photon-gluon
interactions [17].

In this paper, we consider some aspects of the inverse Compton effect, with
particular attention to the polarization effects. We aim to consider the
kinematic conditions providing the maximum cross section at relatively
moderate photon energies, quite achievable in laboratory lasers. An extensive
theoretical material has been accumulated during many years of research into
this elastic interaction of photons with charged particles, which is partly
included in the textbooks. Theoretical description of this process is carried
out within the framework of quantum electrodynamics (QED) and in the lowest
order of perturbation theory is given by two two-vertex diagrams (s- and
t-channel, see for example [18]) In this approximation, the cross section of
the process is calculated analytically. The reader can also find a number of
interesting experimental and theoretical details in reviews [19, 15, 20].

In the paper we mainly use the atomic system of units: $m_{e}=\hbar=|e|=1$. In
these units, the speed of light is $c=137$ and the fine structure constant
$\alpha=1/c$, the classical electron radius $r_{0}=\alpha^{2}$.

## THEORY

### COMPTON’S FORMULA FOR A MOVING TARGET

The theory of the inverse Compton effect is formulated mainly in the framework
of relativistic quantum electrodynamics. Conservation of energy-momentum in
the interaction of a photon with a relativistic charged particle has the form:

$E_{i}+\omega_{i}=E_{f}+\omega_{f},$ $None$ $\vec{p}_{f}=\vec{p}_{i}+\vec{Q}.$
$None$

In (1) $\vec{Q}=\vec{k}_{i}-\vec{k}_{f}$ is the momentum transfer,
$E=\sqrt{p^{2}c^{2}+m^{2}c^{4}}$ is the energy of the particle (electron),
$\omega=ck$ is the photon energy (frequency), and $\vec{k}$ is its momentum.
Substituting the expression for momentum $\vec{p}_{f}$ from (1.2) into $E_{f}$
and using (1.1), we obtain after simple calculations

$\omega_{f}=\omega_{i}\
\frac{\gamma-\cos\theta_{i}\sqrt{\gamma^{2}-1}}{\gamma-\cos\theta_{f}\sqrt{\gamma^{2}-1}+(\omega_{i}/mc^{2})(1-\cos\theta_{fi})}.$
$None$

In (2) $\gamma=E_{i}/mc^{2}$, the particle momentum $\vec{p}_{i}$ is directed
along the $z$ axis, $\theta_{i}$ is the angle between the vectors
$\vec{p}_{i}$ and $\vec{k}_{i}$, $\theta_{f}$ is the angle between the vectors
$\vec{p}_{i}$ and $\vec{k}_{f}$, and $\theta_{fi}$ is the angle between the
vectors $\vec{k}_{i}$ and $\vec{k}_{f}$. At last,
$\cos\theta_{fi}=\cos\theta_{i}\cos\theta_{f}+\sin\theta_{i}\sin\theta_{f}\cos\Phi$.
Next, we use the relation $\vec{p}=\vec{v}(E/c^{2})$ and get
$\gamma=1/\sqrt{1-\beta^{2}}$, where $\beta=v/c<1$ denotes the ratio of the
speed of the particle to the speed of light, i.e. $\gamma$ is the standard
Lorentz-factor. Finally

$\omega_{f}=\omega_{i}\
\frac{(1-\beta\cos\theta_{i})}{(1-\beta\cos\theta_{f})+\sqrt{1-\beta^{2}}\
(\omega_{i}/mc^{2})(1-\cos\theta_{fi})}.$ $None$

For $v=0$, we obtain the Compton formula for the scattering of a photon by a
target at rest. For the inverse Compton scattering, it is usually assumed that
$\theta_{i}=\pi$ and

$\omega_{f}=\omega_{i}\
\frac{(1+\beta)}{(1-\beta\cos\theta_{f})+\sqrt{1-\beta^{2}}\
(\omega_{i}/mc^{2})(1+\cos\theta_{f})}.$ $None$

The photon frequency ratio for this case is shown in Fig. 1.

If in this case the scattered photon moves forward along the axis $z$, then
$\theta_{f}=0$, and

$\omega_{f}=\omega_{i}\ \frac{(1+\beta)}{(1-\beta)+\sqrt{1-\beta^{2}}\
(2\omega_{i}/mc^{2})}.$ $None$

In the ultrarelativistic limit $\beta\to 1$, the photon energy $\omega_{f}$
can formally be arbitrarily large. In this case, the final frequency, as
follows from (5), becomes independent of the initial frequency, and

$\omega_{f}\approx\frac{mc^{2}}{\sqrt{1-\beta^{2}}}=E_{i}.$ $None$

This formula remains valid for small deviations of the angle $\theta_{f}$ from
zero. Moreover, from (1.1) follows $E_{f}\approx\omega_{i}$. It is clear that
in the ultrarelativistic case we have a large energy transfer to the initial
photon. Hence, in particular, it follows that the use of electrons is
energetically more favorable than protons, because at the same energy of
proton and electron, the photon will receive less energy when colliding with
proton due to its larger mass.

Figure 1: The ratio of photon frequencies depending on the scattering angle
$\theta_{f}$ for different values of $\beta$ (4). The energy of the incident
photon is $\omega_{i}=150$ keV.

### DIFFERENTIAL CROSS SECTION

The differential cross section for the scattering of the final photon into a
solid angle $\Omega_{f}$ for unpolarized target particles and photons can be
taken from monograph [18] and, when written in atomic units for an arbitrary
particle mass $m$, takes the form

$\frac{d\sigma}{d\Omega_{f}}=2r^{2}_{0}\left(\frac{\omega_{f}}{mc^{2}\kappa_{1}}\right)^{2}U_{0},$
$None$

where

$U_{0}=4\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)^{2}-4\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)-\left(\frac{\kappa_{1}}{\kappa_{2}}+\frac{\kappa_{2}}{\kappa_{1}}\right),$
$None$
$\kappa_{1}=\frac{2}{m^{2}c^{4}}(E_{i}\omega_{i}-c^{2}p_{i}k_{i}\cos\theta_{i})=2\left(\frac{E_{i}}{mc^{2}}\right)\left(\frac{\omega_{i}}{mc^{2}}\right)(1-\beta\cos\theta_{i}),$
$\kappa_{2}=-\frac{2}{m^{2}c^{4}}(E_{i}\omega_{f}-c^{2}p_{i}k_{f}\cos\theta_{f})=-2\left(\frac{E_{i}}{mc^{2}}\right)\left(\frac{\omega_{f}}{mc^{2}}\right)(1-\beta\cos\theta_{f}).$

For the following calculations we put $\cos\theta_{i}=-1$ and
$\cos\theta_{if}=-\cos\theta_{f}$. We recall that

$\frac{E_{i}}{mc^{2}}=\frac{1}{\sqrt{1-\beta^{2}}}.$

and $\omega_{f}$ is related to $\omega_{i}$ by formula (3). Some examples of
the cross sections are shown in Fig. 2 (left panel). It can be seen from the
figure that, as the target speed increases, the graph of the cross section
squeezes to small scattering angles, which is well known and quite expected.

The formula for cross section (6.1) has the axial symmetry about the $z$ axis,
which is related to the choice of the angle of incidence of the photons
$\theta_{i}=\pi$, however, one can consider other values of the angle of
incidence. Here we calculate cross section (6.1) averaged over the angle
$\Phi$ entering Eq.(3):

$\overline{\frac{d\sigma}{d\Omega_{f}}}=\frac{1}{2\pi r_{0}^{2}}\
\int_{0}^{2\pi}\frac{d\sigma}{d\Omega_{f}}d\Phi,$ $None$

The results of the calculations for the angle $\theta_{i}=90^{\circ}$ are
shown in Fig. 2 (right panel). The averaged cross sections (7) give
practically constant small values in the range of angles $\theta_{f}\geq
90^{\circ}$, and as $\beta$ increases, each subsequent curve decreases faster
than the previous one, which we also see in the left panel. An increase (a
significant on) is observed for the angles $\theta_{f}\sim 0$ with increasing
electron velocities. The general structure of the curves in the figures at
small scattering angles is approximately the same, but the cross section for
$\beta=0.9$ on the right panel is about 2 times larger (!).

Figure 2: Left panel: Differential cross section (6.1) depending on the
scattering angle $\theta_{f}$ for different values of $\beta$. The incident
photon energy is $\omega_{i}=150$ keV, the angle is $\theta_{i}=180^{\circ}$
(head-on collision). Right panel: Averaged differential cross section (7),
calculated taking into account Eqs. (6.1) and (3), $\theta_{i}=90^{\circ}$

This effect can be partly explained as follows. We expand cross section (7)
into a series in Legendre polynomials $P_{l}(\cos\theta_{fi})$ and use the
well-known formula

$P_{l}(\cos\theta_{fi})=P_{l}(\cos\theta_{f})P_{l}(\cos\theta_{i})+2\sum_{m=1}^{l}\frac{(l-m)!}{(l+m)!}\
P_{l}^{m}(\cos\theta_{f})P_{l}^{m}(\cos\theta_{i})\cos m\Phi.$

Integration with respect to the angle $\Phi$ removes the sum, and only the
first term remains in the expansion. If the angle $\theta_{i}\sim
180^{\circ}$, then $P_{l}(-1)=\pm 1$ depending on the parity of $l$. If the
angle $\theta_{i}\sim 90^{\circ}$, then the odd polynomials $P_{l}(0)=0$,
while the even polynomials are not equal to 1. From a mathematical point of
view, the difference in cross sections is explainable, but we still have no
physical explanation.

A completely surprising situation with the cross section arises when
$\theta_{i}=\theta_{f}=0$, i.e. the initial photons fly collinearly the
electron beam and interact with it. The frequency of photons in this case does
not change, but cross section (6.1) for forward scattering is equal to

$\frac{d\sigma}{d\Omega_{f}}=r^{2}_{0}\frac{1+\beta}{1-\beta},$

and grows infinitely in the ultrarelativistic case. This behavior is due to
tending to zero with increasing energy of the denominator of the electron
propagator between two vertices of the Feynman diagram, when the momenta of
the photon and electron are parallel.

For a more detailed study of this interesting phenomenon, we plotted the
dependence of differential cross section (6) on the angle of incidence of the
photon $\theta_{i}$ on a moving electron for a fixed scattering angle
$\theta_{f}=0$, which is presented in Fig. 3. The dependence on the angle
$\Phi$ is absent in this case. In addition, in Fig. 3 we have presented, for
convenience, graphs of frequency ratio $\omega_{f}/\omega_{i}$. From Fig. 3 it
follows, that the cross section is practically constant at low electron
velocities $\beta$ and the change in frequency is small. As $\beta$ grows,
there appears a maximum in the cross section, which shifts to smaller angles
of incidence of the photon with increasing $\beta$. At the same time, the
change in frequency close to the maximum is, of course, significantly smaller
than at the angle $\theta_{i}=180^{\circ}$.

This phenomenon is an unexpected and curious one, and has no direct relation
to the inverse Compton effect with its goal to maximize the frequency of the
final photon. However, we have found it to be interesting and tried to explain
by carrying out a preliminary investigation of the derivative of cross section
(6.1) with respect to $\cos\theta_{i}$ depending on $\beta$. It turned out
that the maximum is already formed at $\beta\approx 0.2$ for
$\cos\theta_{i}\sim-1$, i.e. for the geometry of backscattering. As $\beta$
increases, the value of $\cos\theta_{i}^{max}$ corresponding to the maximum of
the cross section grows quite sharply, crosses zero at $\beta\approx 0.4$ and
tends to one at asymptotic (ultrarelativistic) electron energies, and in this
case $\cos\theta_{i}^{max}(\beta)\approx\beta$. We have obtained this result
for the photon energy $\omega_{i}=150$ keV. A more complete study of this
curious phenomenon requires consideration of other cases.

Figure 3: Differential cross section (6.1) calculated with taking into account
formulas (6.2) and (3) depending on the angle of incidence of the photon
$\theta_{i}$ for different $\beta$. The scattering angle is fixed,
$\theta_{f}=0^{\circ}$. The incident photon energy is $\omega_{i}=150$ keV.
Black (solid) lines show cross sections. For convenience, the red (dashed)
lines show the ratio $\omega_{f}/\omega_{i}$.

### POLARISATION EFFECTS

Let us now consider the influence of the polarizations of colliding particles
on the cross section. In describing polarization effects, we use again the
formulas from monograph [18]. We consider first the most common case of
scattering of polarized photons at unpolarized electrons, where the
polarizations of the final particles are not measured. The scattering plane of
photons is given by the $(z,y)$ axes, and the $x$ axis is orthogonal to this
plane. The degree of linear polarization is characterized by the Stokes
parameter (the index $i$ refers to the initial photon or electron)
$\xi_{3}^{(i)}=\cos 2\alpha$, where the angle $\alpha$ determines the
polarization of the photon with respect to the $x$ axis. The photons polarized
transverse to the scattering plane are characterized by the angle $\alpha=0$,
in the scattering plane $\alpha=\pi/2$. The angle $\alpha=\pi/4$ corresponds
to the absence of linear polarization.

In this case the cross section is also given by Eq. (6.1), where, however,

$U_{0}(\xi_{3}^{(i)})=\left[4\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)^{2}-4\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\right](1-\xi_{3}^{(i)})-\left(\frac{\kappa_{1}}{\kappa_{2}}+\frac{\kappa_{2}}{\kappa_{1}}\right),$
$None$

Figure 4: Differential cross section (6.1) taking into account (8) depending
on the scattering angle $\theta_{f}$ for different values of $\xi_{3}^{(i)}$.
$I:\xi_{3}^{(i)}=1$; $II:\xi_{3}^{(i)}=0.5$; $III:\xi_{3}^{(i)}=0$;
$IV:\xi_{3}^{(i)}=-0.5$; $V:\xi_{3}^{(i)}=-1$ . $\beta=0.7$. The incident
photon energy is $\omega_{i}=150$ keV.

Examples of such cross sections are shown in Fig. 4. From the figure it
follows that the largest cross section is achieved, when the photon is
linearly polarized in the scattering plane, and, in this case, the cross
section is much larger than for the other polarizations.

Next, we consider the scattering of a partially polarized photon by a
partially polarized electron. The covariant form of the cross section in the
approximation of interest to us looks as follows (since we do not consider the
polarizations of the scattered particles, the cross section is multiplied by
the factor of four):

$\frac{d\sigma}{d\Omega_{f}}=2r^{2}_{0}\left(\frac{\omega_{f}}{mc^{2}\kappa_{1}}\right)^{2}[U_{0}(\xi_{3}^{(i)})+(f_{2})_{\mu}\xi^{(i)}_{2}s_{\mu}^{(i)}]$
$None$

The first term is presented explicitly in formula (8) and has been considered
above, the second term describes the scattering of partially circularly
polarized photons by polarized electrons. The 4-vector $f_{2}$ entering it
looks like:

$(f_{2})_{\mu}=\frac{2}{mc}\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\left[k_{i\mu}+k_{f\mu}-2\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\
k_{i\mu}\right],$ $None$

where $k_{\mu}=(\omega/c,\vec{k})$. It is necessary to remark that formula
(10) is similar to formula (26.7.5) in monograph [18], which is misspelled and
differs from formula (10) in the common sign that is due to the use of archaic
notations in this monograph for the coordinates in Minkowski space with
imaginary time component. In later editions of the monograph, a consideration
of polarization effects involving a moving electron is absent at all, as well
as in the most publications that we have looked through.

The Stokes parameter $\xi^{(i)}_{2}$ describes a partially circular
polarization, and the components of the spin 4-vector of a moving electrons
have the form

$s_{0}=\frac{1}{mc}(\vec{p}\cdot\vec{\zeta}),\quad\vec{s}=\vec{\zeta}+\vec{p}\frac{(\vec{p}\cdot\vec{\zeta})}{mc(\sqrt{p^{2}+(mc)^{2}}+mc)}.$
$None$

This formula gives the spin 4-vector $s$ in terms of the 3-dimensional
polarization vector $\vec{\zeta}$ in the rest frame, which is convenient to be
used for the parametrization of the spin vector.

Figure 5: Differential cross section (9) depending on the scattering angle
$\theta_{f}$ for different values of the helicities of colliding particles for
$\xi_{3}^{(i)}=0$ and several values of $\beta$. The incident photon energy is
$\omega_{i}=150$ keV, its angle of incidence is $\theta_{i}=180^{\circ}$. Left
panel: $\quad\xi_{2}^{(i)}\cdot\zeta^{(i)}_{\parallel}=1$; right panel:
$\quad\xi_{2}^{(i)}\cdot\zeta^{(i)}_{\parallel}=-1$.

The general formula for the third term in (9) turns out to be quite
cumbersome, so we will separately consider the longitudinal and transverse
polarizations of the electron. For the transverse polarization

$s_{0}^{(i)}=0,\quad\vec{s}^{(i)}=\vec{\zeta}^{(i)}_{\perp},\quad(\vec{p}_{i}\cdot\vec{\zeta}^{(i)}_{\perp})=0,\quad|\vec{\zeta}^{(i)}_{\perp}|\leq
1,$

and for the longitudinal one

$s_{0}^{(i)}=\frac{|\vec{p}_{i}|}{mc}\,\zeta^{(i)}_{\parallel},\quad\vec{s}^{(i)}=\frac{\vec{p}_{i}E_{i}}{|\vec{p}_{i}|mc^{2}}\,\zeta^{(i)}_{\parallel},\quad|\zeta^{(i)}_{\parallel}|\leq
1.$

Besides that,

$(\vec{k}_{i}\cdot\vec{p}_{i})=k_{i}p_{i}\cos\theta_{i},\quad(\vec{k}_{f}\cdot\vec{p}_{i})=k_{f}p_{i}\cos\theta_{f},$
$(\vec{k}_{i}\cdot\vec{\zeta}^{(i)}_{\perp})=\frac{1}{c}\ \omega_{i}\
\zeta^{(i)}_{\perp}\sin\theta_{i},\quad(\vec{k}_{f}\cdot\vec{\zeta}^{(i)}_{\perp})=\frac{1}{c}\
\omega_{f}\ \zeta^{(i)}_{\perp}\sin\theta_{f},$

The last two formulas are written for the case, where the electron spin vector
lies in the scattering plane, and $\zeta^{(i)}_{\perp}$ can take both positive
and negative values.

As a result, for the transverse polarization of the electron in the scattering
plane, after some transformations, the general formula can be rewritten as
follows:

$((f_{2})_{\mu}\xi^{(i)}_{2}s_{\mu}^{(i)})_{\perp}=-\xi^{(i)}_{2}\zeta^{(i)}_{\perp}\left(\frac{2\omega_{i}}{mc^{2}}\right)\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\
\times$
$\left[\sin\theta_{i}\left(1-2\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\right)+\frac{\omega_{f}}{\omega_{i}}\sin\theta_{f}\right],$
$None$

and for the longitudinal polarization

$((f_{2})_{\mu}\xi^{(i)}_{2}s_{\mu}^{(i)})_{\parallel}=\xi^{(i)}_{2}\zeta^{(i)}_{\parallel}\left(\frac{2\omega_{i}}{mc^{2}\sqrt{1-\beta^{2}}}\right)\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\
\times$
$\left[(\beta-\cos\theta_{i})\left(1-2\left(\frac{1}{\kappa_{1}}+\frac{1}{\kappa_{2}}\right)\right)+\frac{\omega_{f}}{\omega_{i}}(\beta-\cos\theta_{f})\right].$
$None$

From Eq. (12.1), in particular, it follows that if $\theta_{i}=\pi$ and
$\theta_{f}=0$, then the transverse polarization of the electron does not
contribute to the cross section at all.

Figure 6: Differential cross section (9) depending on the scattering angle
$\theta_{f}$ for different values of the polarizations of colliding particles:
$\xi_{3}^{(i)}=0,\quad\xi_{2}^{(i)}=-1,\quad\zeta^{(i)}_{\perp}=1$, and for
several values of $\beta$. The incident photon energy is $\omega_{i}=150$ keV,
its angle of incidence is $\theta_{i}=180^{\circ}$.

Figs. 5,6 present examples of calculating differential cross section (9),
where the linear polarization of the photon $\xi^{(i)}_{3}=0$. In Fig. 5 one
sees the cross sections for the same and opposite helicities of photons and
electrons. Because the polarization parameters enter cross section (12.2) as a
product, then the cases $\xi^{(i)}_{2}\cdot\zeta^{(i)}_{\parallel}=\pm 1$
correspond to the same (opposite) helicities. Comparing Fig. 5 and 2 (left
panel), we see that the electron helicity has a noticeable effect on the value
of the cross section, and the maximum contribution to the cross section for
the forward photon scattering is given by the polarization term with opposite
photon and electron helicities. On the contrary, the same helicities of the
particles noticeably reduce the cross sections.

As for the transverse polarization of the electron (Fig. 6), the contribution
of the term $((f_{2})_{\mu}\xi^{(i)}_{2}s_{\mu}^{(i)})_{\perp}$ to cross
section (9) is very small, and it changes very little the cross section
compared to the unpolarized colliding particles shown in the left panel of
Fig. 2. As a consequence, the sign of the photon helicity $\xi^{(i)}_{2}$ does
not matter.

## CONCLUSION

We have considered some kinematic conditions, including the polarizations of
the colliding electron and photon, at which the maximum values of the
differential cross section are reached at given electron and photon energies.
It has been found that the cross section increases noticeably in the case of
the electron spin being directed along its velocity and the left photon
helicity, or vice versa.

Moreover, an unexpected effect of an increase in the cross section has been
found for angles of incidence of photons on an electron beam different from
$\pi$, while the final photons move along the beam. It has been shown that, as
the angle of incidence of the photons decreases, the cross section increases
and reaches a peak, the value of which increases with the growth of the
electron energy, although such kinematics is not directly related to the
classical inverse Compton effect.

### ACKNOWLEDGEMENTS

The work was carried out with financial support of the Ministry of Science and
higher education of the Russian Federation, project No.075-15-2021-1353.

## References

  * [1] ${\rm https://minobrnauki.gov.ru/grants/grants/?ELEMENT_{I}D=34084}$
  * [2] Compton A.H. // Phys. Rev. 1923. 21. P.483. https://doi.org/10.1103/PhysRev.21.483
  * [3] DuMond J.W.M.// Phys. Rev. 1929. 33. P.643. https://doi.org/10.1103/PhysRev.33.643
  * [4] Kircher M. et al. // Nature Phys. 2020.16. P.756. https://doi.org/10.1038/s41567-020-0880-2
  * [5] Houamer S., Chuluunbaatar O., Volobuev I.P., Popov Yu.V. // Eur.Phys.Journ.D. 2020. 74. P.81. https://doi.org/10.1140/epjd/e2020-100572-1
  * [6] Stepantsov I.S., Volobuev I.P., Popov Yu.V. // Moscow University Physics Bulletin (in russian). 2023 78(1), P. 2310404. https://doi.org/10.55959/MSU0579-9392.78.2310404
  * [7] Burtelov V.A., Kudryashov A.V., Sheshin E.P., Majmaa H.K.K. // Moscow Institute of Physics and Technology Proceedings (in russian). 2019 11(2), P. 116.
  * [8] Artyukov I.A., Vinogradov A.V., Feshchenko R.M. // Physical Basis of Instrumentation (in russian). 2016 5(3), P. 56. https://doi.org/10.25210/jfop-1603-056069
  * [9] Miscellany // Compton effect on the nucleon at low and medium energies. The P.N. Lebedev Physical Institute Proceedings (in russian). 1968 41, Moscow: Nauka.
  * [10] Baranov P.S., Fil’kov L.V., Sokol G.A. // Fortschritte der Physik 1968, 16(11-12), P.595. https://doi.org/10.1002/prop.19680161102
  * [11] Ginzburg V.L., Syrovatskii S.I.// JETP 1964 19(5), P. 1255.
  * [12] Jones F.C. // Phys. Rev. 1968 167, P. 1159. https://doi.org/10.1103/PhysRev.167.1159
  * [13] Blumenthal G.R., Gould R.J.// Rev. Mod. Phys. 1970 42(2), P. 237. https://doi.org/10.1103/RevModPhys.42.237
  * [14] Berezinskii V.S., Bulanov S.V., Dogiel V.A., Ptuskin V.S.// Astrophysics of cosmic rays. Amsterdam: North-Holland (edited by Ginzburg V.L.) 1990, 360 pp.
  * [15] Fargion D, Salis A  // Phys. Usp. 1998 41, P. 823. https://doi.org/10.1070/PU1998v041n08ABEH000432
  * [16] Hütt M.-Th., L’vov A.I., Milstein A.I., Schumacher M. // Physics Reports 2000, 323(6), P.457. https://doi.org/10.1016/S0370-1573(99)00041-1
  * [17] STAR Collaboration, // Sci. Adv. 2023 9(1), P. 3903. https://doi.org/10.1126/sciadv.abq3903
  * [18] Akhiezer A.I., Berestetskii V.B. // Quantum Electrodynamics (in russian). Moscow: Nauka, 1969, 623 pp., §§ 26-27.
  * [19] Kurnosova L.V. (in russian) // UFN 1954 LII (4), P. 603. https://doi.org/10.3367/UFNr.0052.195404c.0603
  * [20] Nedorezov V.G., Turinge A.A., Shatunov Yu.M.// Physics Uspekhi 2004 47(4), P. 341. https://doi.org/10.1070/PU2004v047n04ABEH001743