# Explain then Rank: Scale Calibration of Neural Rankers Using Natural
Language Explanations from Large Language Models

Puxuan Yu<EMAIL_ADDRESS>University of Massachusetts AmherstAmherstMAUSA ,
Daniel Cohen<EMAIL_ADDRESS>Dataminr, Inc.New YorkNYUSA ,  Hemank
Lamba<EMAIL_ADDRESS>Dataminr, Inc.New YorkNYUSA ,  Joel Tetreault
<EMAIL_ADDRESS>Dataminr, Inc.New YorkNYUSA and  Alex Jaimes
<EMAIL_ADDRESS>Dataminr, Inc.New YorkNYUSA

###### Abstract.

The process of scale calibration in ranking systems involves adjusting the
outputs of rankers to correspond with significant qualities like click-through
rates or relevance, crucial for mirroring real-world value and thereby
boosting the system’s effectiveness and reliability. Although there has been
research on calibrated ranking losses within learning-to-rank models, the
particular issue of adjusting the scale for neural rankers, which excel in
handling textual information, has not been thoroughly examined. Neural ranking
models are adept at processing text data, yet the application of existing
scale calibration techniques to these models poses significant challenges due
to their complexity and the intensive training they require, often resulting
in suboptimal outcomes.

This study delves into the potential of large language models (LLMs) to
provide uncertainty measurements for a query and document pair that correlate
with the scale-calibrated scores. By employing Monte Carlo sampling to gauge
relevance probabilities from LLMs and incorporating natural language
explanations (NLEs) to articulate this uncertainty, we carry out comprehensive
tests on two major document ranking datasets. Our findings reveal that the
approach leveraging NLEs outperforms existing calibration methods under
various training scenarios, leading to better calibrated neural rankers.

Scale Calibration, Neural Information Retrieval, Large Language Models,
Natural Language Explanation, Uncertainty

††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information
systems Information retrieval

##  1\. Introduction

Figure 1. The key idea of this study: Neural ranking models struggle to
produce meaningful ranking scores when encountering complex query-document
pairs. We investigate the integration of natural language explanations as
inputs to neural rankers, aiming to simplify the scale-calibrated ranking task
for these rankers.

Neural ranking models act as the core component of many search systems, often
producing the final document scores. However, these scores are usually treated
as transient information and only the relative orderings are preserved to
produce a ranking. While this approach results in well-performing systems with
respect to common retrieval metrics, such as nDCG and MAP, it ignores vital
information used by end users and downstream processes with real-world
impacts, such as fairness (Zerveas et al., 2022) and performance prediction
(Faggioli et al., 2023b; Zamani et al., 2018; Bahri et al., 2020).

The common decision to discard the model scores comes from the fact that
almost all neural ranking models are trained to optimize relative orderings of
documents as opposed to their absolute level of relevance. This notion that
these scores should possess a meaningful real-world interpretation is commonly
referred to as calibration, which ensures that the model’s prediction should
reliably correspond to the true score – a model that has a confidence of
$p=0.47$ should be correct $47\%$ of the time.

Building upon this idea of grounded scores, scale calibration extends this
setting to values that do not have a direct probabilistic interpretation,
which can exist beyond the [0,1] range. This setting is commonly found in the
field of information retrieval (IR) where a ranking model is tasked with
predicting scores for query-document pairs. As discussed, these scores
initially serve to reorder inputs; however, aligning these ranking scores to a
target scale is particularly difficult due to the nature of each query
requiring a differing amount of information to satisfy its information need.
This property is why listwise and pairwise optimizations are so popular for
ranking, as it is much easier to determine if a document is more relevant than
another rather than determining whether the information is sufficient. The
inherent conflict between the objectives of ranking and calibration, where
optimizing for one can compromise the other, is an open problem that we
address in this work. Yan et al. (2022) and Bai et al. (2023) delved into
scale calibration of learning-to-ranking (LTR) models, focusing on the ranking
losses employed during training. However, LTR models are comparatively
lightweight, and necessitate extensive feature engineering, making it
difficult to directly apply their work to larger neural models. This backdrop
frames a critical question for our research: What is the optimal strategy for
scale calibration in neural rankers111Here, “neural rankers” refers to neural
models that produce ranking scores from textual input, distinguishing them
from deep feature-based LTR models.?

To address this question, we first identify several challenges associated with
scale calibration of neural rankers, as opposed to traditional LTR models. We
note that neural rankers often incorporate a backbone neural language model,
such as BERT (Nogueira and Cho, 2019; Nogueira et al., 2019b) and T5 (Nogueira
et al., 2020). This integration results in models that are significantly
larger and more data-hungry compared to LTR models. A second challenge is the
limited availability of training data for calibrated neural rankers. The
process of labeling data on a sophisticated scale, such as with multiple
levels of relevance, demands considerable effort from experienced annotators
or domain experts. Furthermore, the inherent complexity in the textual nature
of queries and documents adds another layer of difficulty. The presence of raw
and intricate information in these texts makes it difficult to learn of an
effective mapping function (i.e., the neural ranker) from the input to the
target scale with limited data.

In light of these challenges, we propose transforming raw textual inputs in a
way that enhances the scale calibration of a neural ranker without
compromising its ranking effectiveness. We aim for these transformation
techniques to be broadly applicable, requiring little to no adaptation across
different tasks and domains. This is accomplished by grounding the text via
external knowledge, i.e leveraging the recent advances in autoregressive large
language models (e.g., GPT-3 (Brown et al., 2020)) and their emerging
capabilities in handling diverse language tasks with minimal supervision (Wei
et al., [n. d.]). Thus, our hypothesis is that the calibrated score of a
query-document pair correlates with the confidence and uncertainty of a large,
zero-shot LLM’s explaination of their relevance. We investigate this
hypothesis via Monte Carlo natural language explanations of query-document
relevance as a means to capture information critical to producing a calibrated
ranking score (see Fig. 1). Specifically, we develop two types of natural
language explanations - one that directly follows the LLM’s own prediction,
and the other explains both assumptions of relevance and irrelevance.

We conduct document ranking experiments on the TREC Deep Learning track
(Craswell et al., 2020) and the NTCIR-14 We Want Web-2 task (Mao et al.,
2019), both containing meaningful multi-level relevance labels. Results
demonstrate that LLM-generated NLEs significantly enhance the scale
calibration of neural rankers, while maintaining or even boosting ranking
performance in most scenarios.

##  2\. Background and Related Work

###  2.1. Scale calibration of learning-to-rank models

Scale calibration is a critical yet relatively unexplored area within
information retrieval. Previous approaches for calibrating ranking models
predominantly involve post-processing methods (Tagami et al., 2013; Chaudhuri
et al., 2017). These techniques typically learn an additional mapping function
that aligns the ranking scores produced by a trained ranker with the target
scale. Yan et al. (2022) highlights a key issue: commonly used pairwise and
listwise ranking losses result in uncalibrated scores. In response, they
propose several solutions, such as multi-objective training, which integrate
uncalibrated ranking loss with calibrated regression loss, alongside a novel
calibrated ranking loss. Building on this, Bai et al. (2023) contend that the
combination of these multi-objectives might not always be synergistic,
potentially leading to suboptimal trade-offs. Until now, efforts in this field
have predominantly centered around refining loss functions for LTR models. Our
research takes a distinct approach, concentrating on identifying text
representations that are most conducive to the scale calibration of neural
rankers.

###  2.2. Uncertainty calibration of neural ranking models

In contrast to the learning-to-rank setting, large neural ranking models are
often restricted by compute limitations. This constraint results in pairwise
loss functions used in place of more effective listwise losses in order to
train the model. Unfortunately, this results in inherently uncalibrated
scores. To achieve a notion of calibration, Cohen et al. (2021) and Penha and
Hauff (2021) suggest leveraging a Bayesian interpretation of these neural
rankers to capture a distribution of possible scores for each document based
on the internal uncertainty of the model. This approach significantly improves
the usability of the model’s scores for downstream tasks by bounding the
possibility of two documents permuting. However, these approaches fail to
ground the scores on any scale as the uncertainty information only makes sense
when comparing documents. We extend their work to scale calibrated scores
through the means of an external, general knowledge LLM.

###  2.3. Natural Language Explanations

Natural language explanations (NLEs) are an emerging point of research in
natural language processing and information retrievals. On one side, these
explanations can be presented directly to users to elucidate the rationale
behind system decisions, such as in recommendation systems (Li et al., 2020).
More pertinently to this study, NLEs have been employed as intermediaries to
enhance task efficacy. For example, Camburu et al. (2018) demonstrated that
generating NLEs for natural language inference samples before prediction can
boost model performance. Similarly, the Chain-of-Thought prompting method (Wei
et al., 2022) employs a sequence of sentences that represent the reasoning
process from input to output, and thus consistently surpasses standard
prompting techniques in tasks such as arithmetic, commonsense, and symbolic
reasoning. Ye and Durrett (2022) introduced the concept of _factuality_ —
measured by the term overlap ratio between NLEs and their corresponding inputs
— as a feature to calibrate LLM predictions. Wadhwa et al. (2023) investigated
explanation-based rescaling, converting coarse numeric scales (e.g., a basic
Likert scale) into finer gradations (e.g., 1-100). This technique leverages
_annotator-written_ explanations and scoring rubrics to prompt LLMs in
generating rescaled outputs. Our research diverges by concentrating on NLEs
autonomously generated by LLMs, which we use as inputs for neural rankers to
facilitate scale calibration. In information retrieval, NLEs contribute to
enhancing the structure and comprehensibility of search engine results pages
(Yu et al., 2022; Rahimi et al., 2021). The ExaRanker framework (Ferraretto et
al., 2023) employs LLM-generated NLEs as additional training signals for
T5-based neural rankers. Our approach mirrors ExaRanker in acquiring
explanations for query-document relevance through LLM prompting. However, we
uniquely apply NLEs to scale calibration instead.

###  2.4. Calibration of large language models

Recently, various studies have been focused on eliciting well calibrated
confidence scores from large language model responses. Most of the work is
centered on treating LLMs as black-boxes and using various prompting based
techniques to generate the confidence scores. For example, Lin et al. ([n.
d.]) leverage verbalized confidence to elicit not only the answer to the
prompt but also the associated confidence score. Similarly relying on self-
evaluation capabilities of LLMs, Kadavath et al. (2022) prompted LLMs to
produce the likelihood of the yes-or-no answer generated by themselves.
Certain other works rely on training external models to associate confidence
to LLM’s responses (Kuhn et al., 2022; Kadavath et al., 2022; Mielke et al.,
2022). Much recently, Tian et al. (2023) conducted more systematic and
evaluations over wider range of methods to obtain confidence scores from LLMs.
They discovered that LLM-verbalized probabilities are better calibrated than
conditional probabilities across several black-box RLHF tuned LLMs (e.g.
ChatGPT, Claude, etc.). This framework is later extended to include hybrid
methods (i.e. a mix of self-consistency and verbalized confidence) and
evaluated the calibration performance on five different types of datasets
(Xiong et al., 2023). In § 4.3, we introduce two methods that take advantage
of the calibration capability of LLMs to directly elicit calibrated ranking
scores.

##  3\. Natural Language Explanation for Scale Calibration

In this section, we formalize the definition of scale calibration, following
which we give justification for leveraging natural language explanations
(NLEs) as a solution for obtaining well calibrated ranking scores. Next, we
detail two methods for acquiring NLEs and explain the process of using them in
the context of scale calibration.

###  3.1. Problem Statement and Motivation

In ranking problems, we define a scoring function $\phi$ for a given query $q$
and its $n$ associated candidate documents $\\{d^{q}\\}_{1}^{n}$. This
function, denoted as $\phi_{\Phi}(q,\\{d^{q}\\})$ produces a score for each
query-document pair under the given retrieval model parameterized by $\Phi$.
The ideal set of configuration for $\Phi$ is obtained by optimizing an
empirical loss on a query-grouped training dataset
$\mathcal{D}=\\{(\\{d^{q}\\},\\{y^{q}\\})\,|\,q\in Q\\}$. Here, $Q$ represents
the set of training set queries, and $\\{y_{q}\\}$ is corresponding set of
labels for each query $q$. The empirical loss is defined as

(1) $\displaystyle\mathcal{L}(\Phi)=\frac{1}{|Q|}\sum_{q\in
Q}l^{\text{rank}}(\\{y^{q}\\},\phi_{\Phi}(q,\\{d^{q}\\}))$

where $l^{\text{rank}}$ is a ranking loss function for an individual query. In
case of neural ranking, the scoring function is defined by a backbone neural
language model. For instance, using a pretrained BERT checkpoint as $\phi$,
concatenating query and each candidate document with a [SEP] token in between
as the inputs, and leveraging cross entropy loss and pairwise cross entropy
loss as $l^{\text{rank}}$ leads to the development of the widely known
monoBERT and duoBERT models (Nogueira et al., 2019a), respectively. However,
it has been observed that popular pairwise and listwise ranking losses are not
scale calibrated due to their translation-invariant property (Yan et al.,
2022)222Although the cross entropy loss used in monoBERT is scale-calibrated,
but it assumes only binary labels; which is something not observed in real-
world datasets e.g. multiple levels of relevance (Craswell et al., 2020; Mao
et al., 2019). This means that adding a constant to all outputs of $\phi$ does
not alter the loss value. To overcome the issue of scale calibration in
ranking loss, in this paper, we incorporate the calibrated listwise softmax
loss (Yan et al., 2022).

However, we posit that merely employing a calibrated ranking loss for training
is insufficient to fully address the scale calibration problem in neural
rankers. Though calibrated ranking loss works well for LTR models, the high
number of parameters in a modern neural ranking model and complexity of the
input distribution results in a non-trivial application of the above work.

Therefore, the challenge is twofold: there is an imbalance in the parameter-
to-data ratio, and the task itself is inherently more complicated.
Consequently, addressing the scale calibration problem in neural rankers
necessitates a multi-faceted approach that goes beyond the application of
calibrated ranking loss alone.

###  3.2. Scale Calibration via Natural Language Explanations

We propose a novel two-step approach to obtain a scale-calibrated numerical
score from the textual query and document. The first step of this approach is
dedicated to a deeper processing and understanding of the contents and
relationships inherent in the input. The second step employs a neural ranker
to convert the outcomes of the first step into calibrated ranking scores. This
overall strategy is grounded in the recent success of LLMs to establish the
relevance of query-document pairs (Ferraretto et al., 2023) and the
demonstrated efficacy of LLM-generated explanations in various reasoning tasks
(Wei et al., 2022). In addition, zero-shot LLMs demonstrate exceptional
ability to adapt to new tasks and domains with little to no efforts, making
our approach much more generalizable.

Specifically, in the first step, we use an LLM, $g(\cdot)$, to generate
natural language explanations (NLEs), $e^{q}$, for each query-document pair,
and then leverage a neural ranker over _only_ the NLEs. This process can be
formally represented as a decomposition of $\phi$ into:

(2) $\displaystyle\phi_{\Phi}(q,\\{d^{q}\\})$
$\displaystyle=f_{\Theta}(g_{\Psi}(q,\\{d^{q}\\}))$ (3)
$\displaystyle=f_{\Theta}(\\{e^{q}\\})$

where $\Psi$ represents the parameters of the LLM, and $\Theta$ encapsulates
the parameters of the neural ranker. The neural ranker $f(\cdot)$ in this
setup is adapted to accept the NLEs of the original inputs as its new inputs.
Note that when using Eq. 1 to optimize parameters $\Phi=\\{\Theta,\Psi\\}$, we
can optimize the parameters of the LLM $g(\cdot)$ and the neural ranker
$f(\cdot)$. While it is possible to perform full or partial finetuning on the
LLM parameters $\Psi$, for simplicity and considering the limited amount of
training data, we choose to freeze $\Psi$ and only optimize parameters of the
neural ranker $\Theta$.

###  3.3. Acquiring NLEs via LLM Prompting

We investigate two distinct methods for acquiring NLEs with varying
characteristics.

####  3.3.1. Literal Explanation

In our primary approach, we straightforwardly present the query and document
to the LLM and request both a relevance prediction (labeled as “relevant” or
“non-relevant”) and an accompanying explanation. The format of the prompt333We
experimented with different prompts, but stick to this one for clear
performance gains and ease of formatting. we employ is as follows:

“For the following query and document, judge whether they are relevant or non-
relevant, and provide an explanation. Output ‘Relevant’ or ‘Nonrelevant’. Do
not repeat the content of the query or the document. Query: {query} Document:
{document} Output:”

This method parallels the prompt used by Ferraretto et al. (2023) to generate
explanations for query-document relevance for training generative rankers.
However, our approach differs due to its simplicity and broader
generalizability across datasets, utilizing zero-shot prompting in contrast to
their use of a fixed set of 7 examples as demonstrations for few-shot
prompting.

A notable limitation of the literal explanation approach is its susceptibility
to inaccuracies stemming from the LLM’s prediction errors. For instance, if
the LLM incorrectly labels a highly relevant query-document pair as “non-
relevant,” the resulting NLE will not accurately reflect the true relevance
score. To mitigate this, we employ a strategy involving Monte Carlo (MC)
sampling of multiple NLEs for the same input and then forming a single NLE via
an aggregation algorithm AGGR (detailed in § 3.4),

(4) $\displaystyle e^{q}=\texttt{AGGR}(\\{y_{i}\sim g_{\Psi}(y|q,d^{q})\\})$

which aims to diminish the influence of erroneous predictions in the
preference of most likely generation. While we find this method beneficial, it
is important to note that there are still a significant number of instances
where the LLM consistently generates incorrect predictions.

####  3.3.2. Conditional Explanation

In response to instances where the LLM persistently errs in judging the
relevance of an input, we also experiment with a different strategy termed
_conditional explanation_. This method involves prompting the LLM to generate
reasons supporting both the relevance and non-relevance of a given query-
document pair. The explanation thus generated is conditional, based on the
presupposed relevance judgment. This approach is also related to sampling
multiple reasoning paths to enhance the self-consistency capabilities of LLMs
(Wang et al., 2023). The prompt we use for this approach is as follows:

“For the following query and document, explain why they are
{relevant/nonrelevant}. Query: {query} Document: {document} Output:”

In the cases where the LLM is highly confident about the relevance (or
nonrelevance) of an input, and we request an explanation for the opposite
judgement, the LLM may not provide useful explanations. For the majority of
instances, the LLM yields conditional explanations from both perspectives.
These explanations are then used by the neural ranker $f(\cdot)$, which learns
to synthesize them to determine a calibrated score (Eq. 3). This approach
allows for a more nuanced understanding and handling of relevance in scenarios
where the LLM’s initial judgment may be skewed or overly confident.

###  3.4. Aggregating Multiple NLEs

As previously discussed, relying solely on the most probable output generated
by the LLM, particularly in the literal explanation approach, could introduce
bias. To mitigate this issue, we propose sampling multiple generations from
the LLMs. This technique has the potential to uncover new insights, which may
either support the initial conclusion with varied reasoning or present
contrasting viewpoints. We consider both outcomes to be advantageous. In the
former scenario, it leads to a more robust and multi-faceted argument
supporting a specific prediction. And in the latter scenario, the generation
of conflicting information by the LLM partially reveals its uncertainty in
comprehending and assessing the content of the input query-document pair. We
hypothesize that this uncertainty is indicative of the potential to predict a
more calibrated ranking score. In essence, any _novel_ information found in
less probable generations can also be valuable. Such information can be
integrated into the construction of the NLE for the given input, culminating
in what we term a “meta” NLE. This meta NLE then serves as a more
comprehensive and nuanced representation of the query-document relationship,
facilitating a more accurate scale calibration in the ranking process.

Definitions: $x$ is an input prompt containing query $q$ and document $d$.
$\mathcal{E}$ is a sentence splitter. $\mathcal{S}$ is a text similarity
function and $\lambda$ is a similarity threshold. $k_{l}$ and $k_{s}$ are
sampling budgets. $g_{\Psi}(y|x)$ is the conditional output distribution
defined by the LLM $\psi$.

Output: Meta NLE $e$

1 function  _AGGR(_$x,\mathcal{E},\mathcal{S},\lambda,k_{l},k_{s}$_)_:

2 $e\leftarrow\varnothing$

3 for  _$i\in 1,2,\cdots,k_{l}$_ do

$y_{i}\leftarrow y\sim g_{\Psi}(y|x)$

// Sample a new response from LLM

4 for  _$s\in\mathcal{E}(y_{i})$_ do

5 if  _$e\neq\varnothing$ and $\text{max}(\\{\mathcal{S}(s,e_{j});e_{j}\in
e\\})>\lambda$_ then

6 continue

7 else

$e\leftarrow e\cup s$

// Add a novel sentence to meta NLE

8 if  _$|e|\geq k_{s}$_ then

return $e$

// Sampling budget reached

9

10 end if

11

12 end if

13

14 end for

15

16 end for

17 return $e$

18

Algorithm 1 Novelty-based NLE aggregation

In our method, we adopt an iterative approach (Algorithm 1) to sample new
responses that elucidate the input query and documents. This process begins
with an initially empty set of sentences (Line 2). During each iteration, if a
sentence from the newly generated explanation (Line 4-5) introduces novel
information – as determined by its maximum text similarity to the existing
sentences in the set being at or below a predefined threshold – then this
sentence is added to the set (Line 9). This sampling of new explanations
continues until we either reach the predefined maximum number of sampling
iterations (Line 3) or fulfill the limit for the number of sentences in the
meta NLE set (Line 10).

##  4\. Experiments

In this section, we rigorously assess our proposed methods using two real-
world, extensively used document ranking datasets. Our evaluations are
conducted under various settings to address the following key research
questions (RQ):

  1. RQ 1

How do LLM-generated natural language explanations impact the calibration and
ranking performance of neural rankers?

  2. RQ 2

Is there a consistent improvement across different training objectives when
using these explanations?

  3. RQ 3

Can sampling multiple instances of NLEs from the LLM yield empirical
improvements?

  4. RQ 4

What inherent limitations are posed by the LLM in our approaches?

This experimental framework is designed to not only quantify the effectiveness
of LLM-generated NLEs in enhancing neural rankers but also to explore the
broader applicability and potential constraints of our methods.

###  4.1. Data

We evaluate the effectiveness of scale calibration methods developed herein,
alongside established baseline techniques, by employing two widely
acknowledged datasets in IR research: the TREC Deep Learning Track (Craswell
et al., 2021), covering the period from 2019 to 2022, and the NTCIR-14 WWW-2
(Mao et al., 2019). Henceforth, for simplicity, we will refer to these
datasets as TREC and NTCIR respectively. The choice of these datasets is
motivated by their comprehensive multi-level relevance judgments provided by
human annotators and the ample volume of labeled documents for each query.
Specifically, TREC uses passages from the MS MARCO collection (Bajaj et al.,
2018), while NTCIR employs web pages from ClueWeb12
Category-B.444https://lemurproject.org/clueweb12/ For TREC, we partition the
queries from the years 2019 and 2020 for training, use the 2021 queries for
validation, and the 2022 queries for testing. For NTCIR, the queries are
divided into training, validation, and testing sets in a 6:2:2 ratio. For a
comprehensive statistical overview and comparative analysis of these datasets,
refer to Table 1.

Table 1. Statistics of the TREC-DL 2019-2022 and NTCIR-14 WWW-2 Datasets. The lengths of queries and documents are quantified using BERT tokenization. For the NTCIR dataset, documents sourced from ClueWeb have undergone preprocessing to retain only the initial 512 tokens. Metric | TREC-DL | NTCIR-14
---|---|---
# Queries (Train/Val/Test) | 97/53/67 | 48/16/16
Avg. # docs per query | 282.7 | 345.3
Levels of relevance | 4 | 5
Label dist. (low to high) | 58/22/14/6 | 48/23/17/8/3
Avg. query length | 8.0 | 22.0
Avg. doc. length | 70.9 | 493.2

###  4.2. Metrics

Our goal is to devise methods that not only enhance the calibration
performance of neural rankers but also maintain their ranking effectiveness.
Accordingly, we evaluate both calibration and ranking aspects. For ranking
evaluation, we employ the normalized discounted cumulative gain (nDCG), which
accounts for multiple levels of relevance judgment. We report nDCG for the
entire ranked lists and also focus on the top 10 results (nDCG@10).

For calibration effectiveness, mean square error (MSE), as well as the
expected calibration error (ECE) (Guo et al., 2017), a metric initially
developed for _classification_ calibration, are commonly used in related
literature (Yan et al., 2022; Bai et al., 2023; Penha and Hauff, 2021). ECE
measures the discrepancy between a model’s predicted confidence between [0,1]
and its actual accuracy. This is achieved by dividing a model’s predictions
into $M$ equally distributed buckets, $B_{m}$, and then evaluating the
deviation of the model’s confidence from the observed accuracy. For instance,
predictions with confidence around $p=0.6$ should, on average, correspond to a
60% accuracy. Yan et al. (2022) extend ECE to regression calibration, where
predictions $p_{i}$ and labels $y_{i}$ are not limited to the [0,1] range. The
ECE for $n$ samples is formally defined as follows:

(5)
$\displaystyle\text{ECE}=\sum^{M}_{m=1}\frac{|B_{m}|}{n}\Big{|}\frac{1}{B_{m}}\sum_{i\in
B_{m}}\hat{y_{i}}-\frac{1}{B_{m}}\sum_{i\in B_{m}}\hat{p_{i}}\Big{|}$

However, given the highly skewed and unbalanced label distribution in our test
set, as indicated in Table 1, MSE and ECE tends to be biased, especially when
optimizing for calibration. Opting for scores that are near the most frequent
label can result in low errors while having no real world usability. This bias
undermines these metrics’ effectiveness in our context. We will further
illustrate this bias of ECE using reliability diagrams (Murphy and Winkler,
1977) in § 5.1. To tackle this issue, we propose the adoption of a class-
balanced version of ECE (CB-ECE), which allocates equal significance to all
labels (candidate scale values). This approach effectively reduces the
inherent bias present in the standard ECE calculation. The CB-ECE is computed
by initially calculating the ECE for each individual label $k$, referred to as
$ECE_{k}$. This calculation follows the formula in Eq. 5, but it is confined
to samples where $\hat{y_{i}}=k$. Subsequently, the mean of all $ECE_{k}$
values is computed across all candidate labels.

Table 2. Ranking and scale calibration performance of baseline methods and our
approaches on two scale calibration datasets TREC and NTCIR. Note that lower
is better with calibration metrics (CB-ECE, ECE and MSE). Statistically
significant improvements over “Platt Scaling monoBERT” are marked with †.

Collection TREC NTCIR Metric Ranking Calibration Ranking Calibration nDCG
nDCG@10 CB-ECE ECE MSE nDCG nDCG@10 CB-ECE ECE MSE A Uncalibrated monoBERT
0.799 0.494 1.205 0.320 0.773 0.735 0.337 1.757 0.799 1.824 B Post hoc +
monoBERT 0.799 0.494 1.141 0.125 0.684 0.735 0.337 1.624 0.457 1.462 C
Finetune monoBERT 0.776 0.422 1.093 0.221 0.721 0.696 0.268 1.843 0.709 1.874
Finetune BERT 0.738 0.327 1.253 0.266 0.785 0.727 0.285 1.756 0.546 1.416 D
LLM prompting w/ rubrics 0.786 0.457 $1.000^{\dagger}$ 1.246 2.137 0.728 0.328
$\bm{1.294}^{\dagger}$ 1.194 2.773 E Post hoc + MC Sampling LLM 0.790 0.473
1.165 0.145 0.673 0.736 $\bm{0.364}^{\dagger}$ 1.677 0.472 1.540 F Literal
Explanation + BERT $0.815^{\dagger}$ $0.529^{\dagger}$ $0.996^{\dagger}$
$\bm{0.067}^{\dagger}$ $\bm{0.602}^{\dagger}$ $\bm{0.742}$ 0.340
$1.534^{\dagger}$ $0.355^{\dagger}$ $1.330^{\dagger}$ Conditional Explanation
+ BERT $\bm{0.822}^{\dagger}$ $\bm{0.534}^{\dagger}$ $\bm{0.862}^{\dagger}$
0.428 0.832 0.720 0.322 $1.405^{\dagger}$ $\bm{0.257}^{\dagger}$
$\bm{1.290}^{\dagger}$

###  4.3. Competing Methods

As the scale calibration of neural rankers remains largely an unexplored area,
we have developed several methods to establish meaningful baselines that
consist of both novel approaches as well as from past related work. These
methods include assessing finetuned neural rankers without any calibration
(Table 2: A), applying post hoc calibration to these finetuned rankers (Table
2: B), further finetuning the rankers with calibration data (Table 2: C), and
utilizing the calibration capabilities of LLMs (Table 2: D,E, § 2.4). In this
section, we delve into the specifics of each of these methods and contrast
them with our core approaches (Table 2: F).

Category A: Uncalibrated rankers. Our approach here employs a BERT-based
ranker that has been previously finetuned using the MS MARCO dataset (Bajaj et
al., 2018). Despite its strong performance in ranking tasks on TREC (in-
domain) and NTCIR (out-of-domain), this ranker lacks scale calibration. This
is attributed to the binary relevance labels in MS MARCO, which differ from
the multi-level relevance judgments used in TREC and NTCIR.

Category B: Post hoc calibration of finetuned rankers. In this method, we
adjust the output scores of the finetuned rankers using a learnable function.
Following the approach of Yan et al. (2022), we apply Platt scaling (Platt,
2000), adapted for regression calibration. Given the output ranking scores of
the model $\mathbf{s}$, the calibrated scores are computed as
$\mathbf{s^{\prime}}=\exp{(w\mathbf{s}+b)}/2$, where $w$ and $b$ are learnable
parameters. It is important to note that under this method, the parameters of
the finetuned rankers remain fixed; only the parameters $w$ and $b$ are
optimized using the scale calibration data. This approach maintains the
original ranking performance of the ranker, provided the final value of $w$ is
positive.

Category C: Finetuning neural rankers on calibration data. This strategy
involves directly finetuning a BERT-based ranker with query-[SEP]-document
style inputs on scale calibration data. Utilizing a checkpoint already
finetuned on MS MARCO allows for a direct comparison with post hoc calibration
methods (Category B) - they use the same initial checkpoint and training data,
but optimize different parameters. Starting with a general-purpose checkpoint
not specifically finetuned for retrieval tasks sets up a direct comparison
with our NLE-based calibration approaches (Category F) - they share the same
initial weights and training data, but the format of their input data differs
significantly.

Category D: LLM prompting using rubrics as contexts. Inspired by recent
studies investigating the capability of LLMs in rendering relevance judgments
(Zhuang et al., 2023; Thomas et al., 2023; Faggioli et al., 2023a) and
rescaling (Wadhwa et al., 2023), we explore leveraging the scoring rubric as a
contextual guide for LLM prompting. The underlying concept is given the
rubric, the LLM’s output should align with calibrated scores, thereby removing
the need for any post-hoc calibration steps. It is important to note that in
this method, the LLM itself essentially functions as the ranker, without the
integration of an additional neural ranking model. For the TREC dataset, we
adopt the prompt method used by Thomas et al. (2023). Conversely, for NTCIR,
we adapt our approach to incorporate the specific rubric outlined in the task
description of NTCIR-14 WWW-2 (Mao et al., 2019). It is noteworthy that the
NTCIR rubric is defined based on scores provided by two annotators (e.g.,
“relevance=3: One annotator rated as highly relevant, one as relevant”),
focusing on the quantitative synthesis of annotations. In contrast, the TREC
rubric is more qualitatively oriented, emphasizing the explanation of query-
document relationships.

Category E: Monte Carlo (MC) sampling LLM predictions. In this method, we use
the LLM as a zero-shot classifier to determine whether a given query and
document pair is relevant (denoted as 1) or not (denoted as 0). To mitigate
the bias inherent in the most probable generation and to minimize instances of
tied scores, which complicate the derivation of rankings, we sample the LLM’s
responses 20 times for each input and calculate the average of these scores.
Subsequently, we employ Platt scaling to these averaged scores and refine the
parameters using the training set. The prompt we use is similar to that of
Zhuang et al. (2023), with one significant modification: we instruct the LLM
to output either “relevant” or “nonrelevant” in lieu of “yes” or “no”. This
alteration stems from our observation that the LLM exhibits a strong prior
towards generating affirmative responses such as “Yes, I can help you with
this request…”, which could potentially skew the predictions. By specifying
the terms “relevant” and “nonrelevant”, we aim to reduce this bias and achieve
more accurate relevance predictions

Category F: Training NLE-based neural rankers on calibration data. Building
upon our methods for generating and aggregating natural language explanations
(NLEs) for query-document pairs, as discussed in § 3, we proceed to finetune a
BERT model (not finetuned for retrieval555We experimented with using a
retrieval-finetuned BERT to initialize the ranker that takes NLEs as inputs,
but found it to perform significantly worse than general purpose BERT
weights.) to process meta NLEs and yield scale-calibrated ranking scores. In
the scenario of the conditional explanation approach (§ 3.3.2), where each
input is represented with two meta NLEs, one for relevance and one for non-
relevance, our method involves an additional processing step. Specifically, we
concatenate the [CLS] hidden states obtained from encoding both NLEs. This
concatenated representation then feeds into an additional linear layer, which
is responsible for transforming these combined hidden states into a final
ranking score. This approach allows the model to integrate insights from both
relevance perspectives.

Figure 2. Ranking and scale calibration performance of the baseline (neural
ranker taking query and documents) and NLE-based approaches on TREC, using
four different optimization objectives. NLE-based approaches consistently
yield better ranking (left) and calibration (right) performance.

###  4.4. Implementation Details

For all experiments that involve LLM, we employ the LlaMA2-13B-Chat model
(Touvron et al., 2023), hosted locally through vLLM666https://github.com/vllm-
project/vllm and using quantized
weights777https://huggingface.co/TheBloke/Llama-2-13B-chat-AWQ, operated on an
A100 GPU. In this study, our neural rankers are based on BERT (Devlin et al.,
2019). We initialize these rankers with weights fine-tuned on MS MARCO,
referred to as monoBERT888https://huggingface.co/veneres/monobert-msmarco in
Table 2, or with weights without retrieval-oriented fine-tuning, simply
denoted as BERT999https://huggingface.co/bert-base-uncased. For fine-tuning on
scale calibration data, we use the AdamW optimizer (Loshchilov and Hutter,
2019) with a learning rate of $3\times 10^{-6}$. The training is conducted
over a maximum of 10 epochs, selecting the best model based on validation set
loss. To mitigate the impact of randomness due to the limited dataset size in
terms of query numbers, each experiment is run with 5 different random seeds.
The metrics reported are averaged across these five runs and the statistical
significance is determined using t-tests with Bonferroni correction
(Weisstein, [n. d.]) at the 95% confidence level. In relation to the
components and hyper-parameters used in Algorithm 1, we employ ROUGE-L (Lin,
2004) as the text similarity function $\mathcal{S}$ following Quach et al.
(2023). The similarity threshold $\lambda$ is set to 0.35, with the sampling
budget $k_{l}$ (maximum number of responses) fixed at 20 and $k_{s}$ (maximum
number of sentences in the meta NLE) at 30. The repository containing the
source code, processed data, and detailed instructions to facilitate the
reproduction of the results reported in this study is available at
https://github.com/PxYu/LLM-NLE-for-Calibration.

###  4.5. Results and Analysis

RQ1. The central research question of this study is to determine if NLEs
generated by LLMs enhance the calibration and ranking performance of neural
rankers. We present the main evaluation results in Table 2, categorizing each
method according to the classifications established in § 4.3 for clear
distinction.

We find that methods utilizing NLEs yield statistically significant
improvements in scale calibration, exhibiting lower CB-ECE values compared to
post hoc calibrating monoBERT (Category B) and calibration fine-tuning
(Category C) across both datasets. Regarding ranking performance, our NLE-
based methods significantly surpass the baseline “calibration finetune BERT,”
which shares the same training data and initial weights, with improvements of
up to $11.4\%$ in nDCG and $63.3\%$ in nDCG@10 in the TREC dataset.
Intriguingly, these methods even exceed the performance of monoBERT (fine-
tuned on MS MARCO) on TREC, indicating that LLM-generated explanations provide
valuable insights for document differentiation. On the NTCIR dataset, the
conditional explanation approach slightly underperforms monoBERT in terms of
ranking, whereas the literal explanation approach still showcase a significant
performance improvement over monoBERT. In summary, RQ1 can be answered
affirmatively that LLM-generated NLEs significantly enhance the scale
calibration of neural rankers, often maintaining or even boosting ranking
performance in most scenarios.

RQ2. Yan et al. (2022) advocate for addressing the scale calibration of LTR
models through calibrated loss functions. While our contribution diverges by
focusing on generalizable content understanding for a more complex task
(ranking texts versus using features), it is essential to investigate whether
our approach is effective across various optimization methods. To this end, we
evaluate our method trained with mean square error (MSE, i.e., regression
loss), uncalibrated listwise softmax (Softmax), a multi-objective combination
of MSE and Softmax (MultiObj), and calibrated listwise softmax (Calibrated
Softmax), all of which have been explored in previous research. As a baseline,
we use fine-tuning neural rankers with query and document inputs (in contrast
to using NLEs), as this presents the most directly comparable counterpart to
our NLE-based methods. The outcomes of this comparative analysis are
illustrated in Fig. 2. Our findings reveal that, across all four ranking
objectives, the NLE-based approaches consistently surpass the performance of
neural models that process raw text queries and documents. This outcome
underscores the effectiveness of our scale calibration methods, positioning
them as significant contributions distinct from those focused on calibrated
loss functions. Furthermore, the consistent improvement across different
training objectives highlights the versatility and robustness of our proposed
approaches.

Table 3. The effect of different types of natural language explanations and selection strategies on the ranking and scale calibration performance of neural rankers. Explanation | Selection | nDCG | CB-ECE
---|---|---|---
Literal | Most Probable | 0.789 | 1.093
Aggregate MC | 0.815 | 0.996
Oracle | $\bm{0.883}$ | $\bm{0.801}$
Conditional | Most probable | 0.797 | 0.895
Aggregate MC | $\bm{0.822}$ | $\bm{0.862}$

RQ3 and RQ4. To further understand the dynamics between LLM-generated NLEs and
their impact on scale calibration and ranking, we execute additional
experiments on the TREC dataset, examining various NLE types. As advocated in
§ 3.4, we explore the concept of aggregating multiple MC samples of
explanations into a single meta NLE (here referred to as aggregate MC). This
approach aims to reduce bias and incorporate richer reasoning and uncertainty
information. For comparative analysis, we establish a control group where only
the most probable explanation is employed as input for the neural ranker. Our
investigation also extends to understanding the influence of the LLM’s
inherent limitations, particularly in its judgment alignment with human
annotators (Faggioli et al., 2023a; Thomas et al., 2023), on the performance
of downstream NLE-based neural rankers. In our literal explanation approach,
we keep sampling responses from the LLM, containing both prediction and
explanation, until we obtain a response that aligns with the binary relevance
judgment provided by annotators. Specifically, a match occurs when the LLM
predicts ”relevant” for labels 1, 2, or 3, and ”nonrelevant” for label 0. If
alignment is not achieved within 20 samples, we default to using the most
probable, albeit incorrect, explanation. This experimental condition is
denoted as the oracle setting. The results of these experimental setups are
presented in Table 3.

The results clearly demonstrate that the aggregate MC method substantially
surpasses the use of only the most probable explanation in both ranking and
scale calibration, across both literal and conditional explanation setups.
This finding supports our hypothesis regarding the benefits of MC sampling
multiple explanations. It also underscores the effectiveness of our
aggregation strategy, affirming its utility in this context. Moreover, a
notable performance disparity is observed between our current aggregate MC
approach and the oracle explanation setting. This gap highlights a significant
potential for improvement in NLE-based neural rankers, contingent upon better
alignment of LLM judgments with human annotators on a _binary_ scale.
Specifically, enhancements in this alignment could be achieved through methods
such as fine-tuning, prompt engineering, or employing larger LLMs with more
advanced reasoning capabilities. We leave the exploration of this direction
for future work.

Figure 3. Reliability diagrams for two models on TREC: The left diagram shows
a model with ranking scores densely concentrated on the lower part of the
scale, which exhibits better ECE performance due to ECE’s failure to account
for prediction coverage across the target scale. On the right, the CB-ECE
penalizes this undesirable behavior, indicating that the model providing
better coverage across the scale is more effectively calibrated. Table 4. An
example showing natural language explanations generated by the LLM lead to
better calibrated scores. Query: “Example of what a family advocate does”
Document: “This person advocates for the best interest of the child, and is
tasked with investigating the family situation and advising the court what
custody situation would be in the best interests of the child.”
Label: 2 (Highly relevant: The passage has some answer for the query, but the
answer may be a bit unclear, or hidden amongst extraneous information).
---
Approach |  Explanation |  Score
Uncalibrated monoBERT |  N/A |  0.872
Calibrated monoBERT |  N/A |  0.397
Literal
Explanation |  here’s my answer: relevant the query asks for an example of what a family advocate does, and the document provides an example of a family advocate’s role, specifically advocating for the best interests of the child and investigating the family situation to advise the court on the appropriate custody arrangement. this is directly related to the query and the document provides relevant information. the document provides a clear example of what a family advocate does, making it relevant to the query. the document directly addresses the query’s question, making it a relevant match. sure, i can help you with that! |  $\bm{1.980}$
Conditional Explanation |  Relevance: the query “example of what a family advocate does” is relevant to the provided document because the document describes the role of a family advocate, which is to advocate for the best interests of the child in family law cases. …… the document highlights the key aspects of a family advocate this is an example of what a family advocate might do in practice. this connection between the query and document makes them relevant to each other. 3. focus on the child’s best interests: the document emphasizes the importance of prioritizing the child’s best interests when making decisions about child custody.

Non-Relevance: the query “example of what a family advocate does” is not relevant to the document provided because the document does not provide an example of what a family advocate does, but rather describes the responsibilities of a family advocate. the document states that a family advocate advocates for the best interests of the child and investigates the family situation to advise the court on the custody situation that would be in the best interests of the child. this is a description of the role and responsibilities of a family advocate, but it does not provide an example of what a family advocate does in a specific situation. |  $\bm{2.053}$

##  5\. Discussions

###  5.1. Limitations of the ECE Metric

We discuss in § 4.2 the limitations inherent in using the expected calibration
error (ECE) (Guo et al., 2017) for scale calibration data, especially when
faced with imbalanced label distributions. To counter this challenge, we
propose a straightforward modification: the adoption of class-balanced ECE
(CB-ECE). This section uses model outputs on TREC as illustrative examples to
provide detailed insights and empirical support for our approach.

We employ reliability diagrams (Murphy and Winkler, 1977), a widely-used
technique in calibration, to visualize calibration performance. In ECE
calculations, samples are sorted by their predicted values and divided into
several buckets. The calibration error for each bucket is determined by the
absolute difference between the mean labels and mean predictions of samples
within that bucket, further _weighted by_ the number of samples falling in the
bucket. Reliability diagrams concretely visualize this concept, with each
bucket represented by a bar. The mean predictions are indicated on the x-axis,
while the mean labels (i.e., scale) are represented by the height of the bars.
In an ideal scenario, a perfectly calibrated model would yield a reliability
diagram aligning with the diagonal line, indicating that mean predictions
correspond accurately to the labels for every bucket. However, ECE overlooks a
critical aspect of scale calibration: coverage. This refers to the model’s
ability to adequately span the target scale range. Figure 3 illustrates this
point with two diagrams. The left diagram shows a model whose outputs are
narrowly concentrated around lower values on TREC’s 0-3 scale. The right
diagram, conversely, exhibits outputs covering a broader range on the target
scale, closely aligned with the diagonal line. Relying solely on ECE values
would misleadingly suggest that the first model is better calibrated. However,
we argue that a model with predictions confined to a limited portion of the
desired scale, even if skewed towards certain values popular in the evaluation
set, cannot be considered well-calibrated. CB-ECE offers a solution by re-
adjusting the significance of each target scale. In the example shown in Fig.
3, CB-ECE more accurately indicates that the second model exhibits better
scale calibration, which is more in line with our intuitive understanding of
model calibration.

###  5.2. Case Study

We randomly show a pair of query and document from TREC, as well as LLM-
generated explanations for them in order to predict calibrated ranking scores.
The example is shown in Table 4. The superior effectiveness of our methods in
assessing the relevance of the query “example of what a family advocate does”
to a specific document can be attributed to several aspects. Firstly, the use
of NLEs generated by an LLM provides a deeper contextual understanding. For
instance, the first approach’s explanation elucidates the direct relevance by
highlighting how the document exemplifies the role of a family advocate,
particularly in advocating for a child’s best interests. This detailed
breakdown goes beyond mere textual similarity, offering a richer
interpretation of the document’s relevance to the query. Furthermore, our
second approach’s dual perspective in explaining both relevance and non-
relevance brings a comprehensive understanding. It captures the essence of the
document’s role description and contrasts it with the lack of specific
examples, thereby offering a deeper analysis of the document-query
relationship. By feeding these elaborate explanations into BERT, our methods
effectively leverage more informative data, allowing for a more refined and
accurate relevance assessment. Consequently, our methods demonstrate a notable
improvement in capturing the subtleties and depths of relevance, as evidenced
by the scores closer to the label compared to the baseline approaches.

##  6\. Conclusion and Future Works

This study addressed the challenge of scale calibration for neural rankers,
aiming to align ranking scores with meaningful real-world measures. Our
proposed method capitalizes on zero-shot LLMs’ inherent understanding of
textual data to enhance scale calibration, while preserving or boosting
ranking performance. We employed natural language explanations (NLEs)
generated by LLMs for ranking, demonstrating that our approach surpasses
established baseline methods in terms of ranking and calibration metrics and
shows consistency across various training objectives. Additionally, we
outlined strategies to mitigate the inherent uncertainty in LLM outputs
through effective aggregation.

Despite the effectiveness of using zero-shot LLMs, there is potential for
improvement through more sophisticated strategies like few-shot prompting (Wei
et al., [n. d.]), instruction tuning (Zhu et al., 2024), and the use of LLMs
with advanced reasoning capabilities. Future research could also focus on
increasing the efficiency of NLE generation through techniques like
distillation (Gu et al., 2023; Shridhar et al., 2023). Furthermore, enhancing
the reliability of explanations (Ye and Durrett, 2022) represents another
promising avenue for developing better calibrated rankers.

## References

  * (1)
  * Bahri et al. (2020) Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, and Andrew Tomkins. 2020.  Choppy: Cut Transformer for Ranked List Truncation. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, Virtual Event China, 1513–1516.  https://doi.org/10.1145/3397271.3401188
  * Bai et al. (2023) Aijun Bai, Rolf Jagerman, Zhen Qin, Le Yan, Pratyush Kar, Bing-Rong Lin, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2023.  Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_. ACM, Birmingham United Kingdom, 4502–4508.  https://doi.org/10.1145/3583780.3614712
  * Bajaj et al. (2018) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.  https://doi.org/10.48550/arXiv.1611.09268 arXiv:1611.09268 [cs].
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , Vol. 33. Curran Associates, Inc., 1877–1901.  https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-SNLI: Natural Language Inference with Natural Language Explanations. In _Advances in Neural Information Processing Systems_ , Vol. 31. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html
  * Chaudhuri et al. (2017) Sougata Chaudhuri, Abraham Bagherjeiran, and James Liu. 2017.  Ranking and Calibrating Click-Attributed Purchases in Performance Display Advertising. In _Proceedings of the ADKDD’17_. ACM, Halifax NS Canada, 1–6.  https://doi.org/10.1145/3124749.3124755
  * Cohen et al. (2021) Daniel Cohen, Bhaskar Mitra, Oleg Lesota, Navid Rekabsaz, and Carsten Eickhoff. 2021.  Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 654–664.  https://doi.org/10.1145/3404835.3462951 arXiv:2105.04651 [cs].
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020.  Overview of the TREC 2019 deep learning track.  http://arxiv.org/abs/2003.07820 arXiv:2003.07820 [cs].
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M. Voorhees, and Ian Soboroff. 2021.  TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, Virtual Event Canada, 2369–2375.  https://doi.org/10.1145/3404835.3463249
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Faggioli et al. (2023a) Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. 2023a.  Perspectives on Large Language Models for Relevance Judgment. In _Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval_. ACM, Taipei Taiwan, 39–50.  https://doi.org/10.1145/3578337.3605136
  * Faggioli et al. (2023b) Guglielmo Faggioli, Thibault Formal, Stefano Marchesin, Stéphane Clinchant, Nicola Ferro, and Benjamin Piwowarski. 2023b.  Query Performance Prediction for Neural IR: Are We There Yet?  http://arxiv.org/abs/2302.09947 arXiv:2302.09947 [cs].
  * Ferraretto et al. (2023) Fernando Ferraretto, Thiago Laitz, Roberto Lotufo, and Rodrigo Nogueira. 2023.  ExaRanker: Explanation-Augmented Neural Ranker.  https://doi.org/10.48550/arXiv.2301.10521 arXiv:2301.10521 [cs].
  * Gu et al. (2023) Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023.  Knowledge Distillation of Large Language Models.  https://doi.org/10.48550/arXiv.2306.08543 arXiv:2306.08543 [cs].
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On Calibration of Modern Neural Networks.  http://arxiv.org/abs/1706.04599 arXiv:1706.04599 [cs].
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022.  Language Models (Mostly) Know What They Know.  http://arxiv.org/abs/2207.05221 arXiv:2207.05221 [cs].
  * Kuhn et al. (2022) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022.  Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.  https://openreview.net/forum?id=VD-AYtP0dve
  * Li et al. (2020) Lei Li, Yongfeng Zhang, and Li Chen. 2020.  Generate Neural Template Explanations for Recommendation. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_ _(CIKM ’20)_. Association for Computing Machinery, New York, NY, USA, 755–764.  https://doi.org/10.1145/3340531.3411992
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin et al. ([n. d.]) Stephanie Lin, Jacob Hilton, and Owain Evans. [n. d.].  Teaching models to express their uncertainty in words.  ([n. d.]).
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.  http://arxiv.org/abs/1711.05101 arXiv:1711.05101 [cs, math].
  * Mao et al. (2019) Jiaxin Mao, Tetsuya Sakai, Cheng Luo, Peng Xiao, Yiqun Liu, and Zhicheng Dou. 2019.  Overview of the NTCIR-14 We Want Web Task.  (2019).
  * Mielke et al. (2022) Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y.-Lan Boureau. 2022.  Reducing conversational agents’ overconfidence through linguistic calibration.  http://arxiv.org/abs/2012.14983 arXiv:2012.14983 [cs].
  * Murphy and Winkler (1977) Allan H. Murphy and Robert L. Winkler. 1977.  Reliability of Subjective Probability Forecasts of Precipitation and Temperature.  _Journal of the Royal Statistical Society Series C: Applied Statistics_ 26, 1 (March 1977), 41–47.  https://doi.org/10.2307/2346866
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT. In _arXiv:1901.04085 [cs]_.  http://arxiv.org/abs/1901.04085 arXiv: 1901.04085.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document Ranking with a Pretrained Sequence-to-Sequence Model. In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 708–718.  https://doi.org/10.18653/v1/2020.findings-emnlp.63
  * Nogueira et al. (2019a) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019a.  Multi-Stage Document Ranking with BERT.  http://arxiv.org/abs/1910.14424 arXiv:1910.14424 [cs].
  * Nogueira et al. (2019b) Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b.  _Document Expansion by Query Prediction_.  Technical Report arXiv:1904.08375. arXiv.  http://arxiv.org/abs/1904.08375 arXiv:1904.08375 [cs] type: article.
  * Penha and Hauff (2021) Gustavo Penha and Claudia Hauff. 2021.  On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 160–170.  https://doi.org/10.18653/v1/2021.eacl-main.12
  * Platt (2000) John Platt. 2000.  Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods.  _Adv. Large Margin Classif._ 10 (June 2000).
  * Quach et al. (2023) Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. 2023.  Conformal Language Modeling.  http://arxiv.org/abs/2306.10193 arXiv:2306.10193 [cs].
  * Rahimi et al. (2021) Razieh Rahimi, Youngwoo Kim, Hamed Zamani, and James Allan. 2021.  Explaining Documents’ Relevance to Search Queries.  https://doi.org/10.48550/arXiv.2111.01314 arXiv:2111.01314 [cs].
  * Shridhar et al. (2023) Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023.  Distilling Reasoning Capabilities into Smaller Language Models. In _Findings of the Association for Computational Linguistics: ACL 2023_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 7059–7073.  https://doi.org/10.18653/v1/2023.findings-acl.441
  * Tagami et al. (2013) Yukihiro Tagami, Shingo Ono, Koji Yamamoto, Koji Tsukamoto, and Akira Tajima. 2013.  CTR prediction for contextual advertising: learning-to-rank approach. In _Proceedings of the Seventh International Workshop on Data Mining for Online Advertising_. ACM, Chicago Illinois, 1–8.  https://doi.org/10.1145/2501040.2501978
  * Thomas et al. (2023) Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023.  Large language models can accurately predict searcher preferences.  http://arxiv.org/abs/2309.10621 arXiv:2309.10621 [cs].
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023.  Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.  https://doi.org/10.48550/arXiv.2305.14975 arXiv:2305.14975 [cs].
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  http://arxiv.org/abs/2307.09288 arXiv:2307.09288 [cs].
  * Wadhwa et al. (2023) Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. 2023.  Using Natural Language Explanations to Rescale Human Judgments.  https://doi.org/10.48550/arXiv.2305.14770 arXiv:2305.14770 [cs].
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023.  Self-Consistency Improves Chain of Thought Reasoning in Language Models.  http://arxiv.org/abs/2203.11171 arXiv:2203.11171 [cs].
  * Wei et al. ([n. d.]) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raﬀel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, and Donald Metzler. [n. d.].  Emergent Abilities of Large Language Models.  ([n. d.]).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  _Advances in Neural Information Processing Systems_ 35 (Dec. 2022), 24824–24837.  https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html
  * Weisstein ([n. d.]) Eric W. Weisstein. [n. d.].  Bonferroni Correction.  https://mathworld.wolfram.com/ Publisher: Wolfram Research, Inc..
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023.  Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.  https://doi.org/10.48550/arXiv.2306.13063 arXiv:2306.13063 [cs].
  * Yan et al. (2022) Le Yan, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2022.  Scale Calibration of Deep Ranking Models. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. ACM, Washington DC USA, 4300–4309.  https://doi.org/10.1145/3534678.3539072
  * Ye and Durrett (2022) Xi Ye and Greg Durrett. 2022.  The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning.  http://arxiv.org/abs/2205.03401 arXiv:2205.03401 [cs].
  * Yu et al. (2022) Puxuan Yu, Razieh Rahimi, and James Allan. 2022.  Towards Explainable Search Results: A Listwise Explanation Generator. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, Madrid Spain, 669–680.  https://doi.org/10.1145/3477495.3532067
  * Zamani et al. (2018) Hamed Zamani, W. Bruce Croft, and J. Shane Culpepper. 2018.  Neural Query Performance Prediction using Weak Supervision from Multiple Signals. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. ACM, Ann Arbor MI USA, 105–114.  https://doi.org/10.1145/3209978.3210041
  * Zerveas et al. (2022) George Zerveas, Daniel Cohen, Navid Rekabsaz, and Carsten Eickhoff. 2022.  Mitigating Bias in Search Results Through Contextual Document Reranking and Neutrality Regularization.  (2022), 7.
  * Zhu et al. (2024) Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zhicheng Dou, Zheng Liu, and Ji-Rong Wen. 2024.  INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning.  https://doi.org/10.48550/arXiv.2401.06532 arXiv:2401.06532 [cs].
  * Zhuang et al. (2023) Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Berdersky. 2023.  Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels.  http://arxiv.org/abs/2310.14122 arXiv:2310.14122 [cs].


aainstitutetext: Department of Physics and Astronomy, University of
California, Los Angeles, CA 90095, USAbbinstitutetext: Mani L. Bhaumik
Institute for Theoretical Physics, University of California, Los Angeles, CA
90095, USAccinstitutetext: Center for Frontiers in Nuclear Science, Stony
Brook University, Stony Brook, NY 11794, USAddinstitutetext: Center for
Theoretical Physics, Massachusetts Institute of Technology, Cambridge, MA
02139eeinstitutetext: Department of Physics and Center for Field Theory and
Particle Physics, Fudan University, Shanghai, Chinaffinstitutetext: Key
Laboratory of Nuclear Physics and Ion-beam Application (MOE), Fudan
University, Shanghai, China

# Collins-type Energy-Energy Correlators and Nucleon Structure

Zhong-Bo Kang d , Kyle Lee e,f , Ding Yu Shao a,b and Fanyi Zhao
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We generalize the conventional Energy-Energy Correlator (EEC) to include the
azimuthal angle dependence, so to define azimuthal angle dependent EEC
observables. We study this new EEC observable in $e^{+}e^{-}$ and semi-
inclusive deep inelastic scattering (SIDIS). In the back-to-back region, we
find that the azimuthal angle dependent EEC is sensitive to both the
unpolarized EEC jet function and a Collins-type EEC jet function. While the
unpolarized EEC jet function is related to the unpolarized transverse momentum
dependent (TMD) fragmentation function, the Collins-type EEC jet function is
connected with the Collins fragmentation function. We further demonstrate how
the new observables allow us to access to the 3D structure of nucleons,
especially the spin-dependent ones.

##  1 Introduction

The energy-energy correlators (EEC) Basham:1978bw ; Basham:1978zq  is one of
the earliest infrared and collinear (IRC) safe event shape observables
Kinoshita:1962ur ; Lee:1964is . This event shape observable has been studied
with unpolarized scattering beams in electron-positron collisions and Semi-
Inclusive Deep Inelastic Scattering (SIDIS). Various experimental measurements
have been provided for $e^{+}e^{-}$ collisions CELLO:1982rca ; JADE:1984taa ;
Fernandez:1984db ; PLUTO:1985yzc ; TASSO:1987mcs ; Wood:1987uf , which have
revealed insights into the structure and properties of hadrons and other
subatomic particles.

The conventional EEC observable measures the correlation between the energies
of two particles produced in the collision, while they are separated by the
angle $\theta_{ij}$ Moult:2018jzp . In this work, we generalize this
observable to include the azimuthal angle $\phi_{ij}$ between the two
particles and study how this azimuthal angle dependent EEC enables us to go
beyond what is possible with the conventional EEC for probing nucleon
structure, in particular the spin-dependent nucleon structure. We start with
the definition of this observable in $e^{+}e^{-}$ collisions,

$\displaystyle\mathrm{{{EEC}}}_{e^{+}e^{-}}(\tau,\phi)=$
$\displaystyle\frac{1}{2}\sum_{i,j}\int
d\theta_{ij}dz_{i}dz_{j}z_{i}z_{j}\frac{1}{\sigma}\frac{d\sigma}{dPS}\delta\left(\tau-\left(\frac{1+\cos\theta_{ij}}{2}\right)\right)\delta(\phi-\phi_{ij})\,,$
(1)

where $z_{i}$ and $z_{j}$ are the energy fractions of the final-state hadrons
$i,j$, separated by $\theta_{ij}$, angle $\phi_{ij}$ is the azimuthal angle
difference between the two hadrons in the Gottfried-Jackson (GJ) frame
Gottfried:1964nx  as illustrated in the left panel of Fig. 1, and
$dPS=d\theta_{ij}d\phi_{ij}dz_{i}dz_{j}$. In the next section where we study
the EEC in the back-to-back region, we will demonstrate that this azimuthal
angle dependent polarized EEC gives sensitivity to both the traditional EEC
jet function as well as a new EEC jet function, named “Collins-type” EEC jet
function. The traditional EEC jet function has a close relation to the
unpolarized fragmentation functions $D_{1}(z,{k}_{\perp}^{2})$ Moult:2018jzp ,
while the Collins-type EEC jet function is related to the well-known Collins
fragmentation functions $H_{1}^{\perp}(z,{k}_{\perp}^{2})$ toappear . In the
section below, we will also discuss the EEC for the SIDIS.

Figure 1: Illustration of EEC for $e^{+}e^{-}$ annihilation in GJ frame (left)
and for Deep Inelastic Scattering in Breit frame (right).

##  2 EEC in the back-to-back region for $e^{+}e^{-}$ annihilation

In the back-to-back region, one has $\theta_{ij}\to\pi$ and thus $\tau\to 0$
in Eq. (1). It can be shown
$\displaystyle\tau={\bm{P}_{h_{\perp}}^{2}}/({z_{1}^{2}Q^{2}})$ in this
region. Here $\bm{P}_{h_{\perp}}$ is the transverse momentum of the hadron
$h_{1}$ in the so-called Gottfried-Jackson (GJ) frame Gottfried:1964nx ,
namely with respect to hadron $h_{2}$ in the pair production, see Fig. 1. By
introducing $\bm{q}_{T}=-\bm{P}_{h_{\perp}}/z_{1}$, one obtains the observable
$\tau$ represented by $\displaystyle\tau={\bm{q}_{T}^{2}}/{Q^{2}}$.

Using these variables to rewrite Eq. (1), we find that the azimuthal angle
dependent EEC, $\mathrm{{{EEC}}}_{e^{+}e^{-}}(\tau,\phi)$, is related to
$\bm{q}_{T}$-differential cross section as

$\displaystyle\mathrm{{{EEC}}}_{e^{+}e^{-}}(\tau,\phi)$
$\displaystyle=\frac{1}{2}\sum_{i,j}\int
d^{2}\bm{q}_{T}dz_{i}dz_{j}z_{i}z_{j}\frac{1}{\sigma}\frac{d\sigma}{d^{2}\bm{q}_{T}dz_{i}dz_{j}}\delta\left(\tau-\frac{\bm{q}_{T}^{2}}{Q^{2}}\right)\delta(\phi-\phi_{q_{T}})\,.$
(2)

Starting from the azimuthal angle dependent factorization in the GJ frame (see
e.g. in Kang:2015msa ), especially after Fourier transforming from
$\bm{q}_{T}$-space to $\bm{b}$-space, one would end up with the usual
unpolarized EEC jet function ${J}_{q}({\bm{b}},\mu,\zeta)$ Moult:2018jzp  and
the Collins-type EEC jet functions ${J}_{q}^{\perp}({\bm{b}},\mu,\zeta)$

$\displaystyle{J}_{q}({\bm{b}},\mu,\zeta)\equiv$
$\displaystyle\sum_{h}\int_{0}^{1}dz\,z\,\tilde{D}_{1,h/q}(z,\mu_{b_{*}},\zeta_{i})e^{-S_{\rm
pert}(\mu,\mu_{b_{*}})-S_{\rm
NP}^{D_{1}}(b,Q_{0},\zeta)}\left(\sqrt{\frac{\zeta}{\zeta_{i}}}\right)^{\kappa\left(b,\mu_{b_{*}}\right)}\,,$
(3) $\displaystyle{J}_{q}^{\perp}({\bm{b}},\mu,\zeta)\equiv$
$\displaystyle\sum_{h}\int_{0}^{1}dz\,\tilde{H}_{1,h/{q}}^{\perp}(z,\mu_{b_{*}},\zeta_{i})e^{-S_{\rm
pert}(\mu,\mu_{b_{*}})-S_{\rm
NP}^{H^{\perp}_{1}}(b,Q_{0},\zeta)}\left(\sqrt{\frac{\zeta}{\zeta_{i}}}\right)^{\kappa\left(b,\mu_{b_{*}}\right)}\,,$
(4)

where $\tilde{D}_{1,h/q}$ and $\tilde{H}_{1,h/{q}}^{\perp}$ are related to
unpolarized TMD FFs and Collins functions respectively. At the end of the day,
we obtain the $\phi$-dependent EEC,

$\displaystyle\mathrm{{{EEC}}}_{e^{+}e^{-}}(\tau,\phi)=$
$\displaystyle\,\frac{2\pi N_{c}\alpha_{\rm
em}^{2}}{3Q^{2}\sigma}\sum_{q}e_{q}^{2}\int
d{\bm{q}}_{T}^{2}\delta(\tau-\frac{{\bm{q}}_{T}^{2}}{Q^{2}})\left(Z_{uu}+\cos
2\phi\,Z_{\rm collins}\right)\,,$ (5)

where we have

$\displaystyle Z_{uu}=$
$\displaystyle\int\frac{bdb}{2\pi}J_{0}(bq_{T}){J}_{q}({\bm{b}},\mu,\zeta){J}_{\bar{q}}({\bm{b}},\mu,\zeta)\,$
(6) $\displaystyle Z_{\rm collins}=$
$\displaystyle\,\int\frac{bdb}{2\pi}\frac{b^{2}}{8}J_{2}(bq_{T}){J}_{q}^{\perp}({\bm{b}},\mu,\zeta){J}_{\bar{q}}^{\perp}({\bm{b}},\mu,\zeta)$
(7)

For a phenomenology example, we study the ratio $A_{ee}(\tau)={Z_{\rm
collins}}\big{/}{Z_{uu}}$. Moreover, by substituting the summation over all
the final hadrons $h$ in Eqs. (3) and (4) to a subset $\mathbb{S}$ for each
direction in the pair production, one has EEC of subsets
$\mathbb{S}\times\mathbb{S}$. As pions are the primary measurements available
for the Collins-type asymmetry, we now study the ratio
$A_{ee}^{\mathbb{S}\times\mathbb{S}}(\tau)$ with different combinations of
pion pairs. In Fig. 2, we plot the Collins asymmetry for subsets
$\mathbb{S}=\\{\pi^{+}\\},\ \\{\pi^{-}\\}$ or $\\{\pi^{\pm}\\}$ using TMD FFs
$D_{1}(z,k_{\perp}^{2})$ and Collins function $H_{1}^{\perp}(z,k_{\perp}^{2})$
extracted in Kang:2015msa . When choosing a subset of either positively or
negatively charged pions detected in EEC, one observes sizable asymmetries,
which worth further measurement.

Figure 2: $A_{ee}^{\mathbb{S}\times\mathbb{S}}(\tau)$ for
$\mathbb{S}\times\mathbb{S}=\\{\pi^{+},\ \pi^{-}\\}\times\\{\pi^{+},\
\pi^{-}\\}$, $\\{\pi^{+}\\}\times\\{\pi^{-}\\}$ and
$\\{\pi^{-}\\}\times\\{\pi^{-}\\}$ at $\sqrt{s}=10.6$ GeV with Collins
functions fitted in Kang:2015msa .

##  3 EEC in the back-to-back region for SIDIS

SIDIS is a fundamental tool in the study of the structure of nucleons and
nuclei. For the process

$\displaystyle e^{-}(l)+p(P)\rightarrow e^{-}(l^{\prime})+h(P_{h})+X\,,$ (8)

the SIDIS version of the EEC that will be generalized to EECDIS here was
developed recently Li:2021txc , and was characterized as a function of the
angular separation $\theta_{ap}$ between the incoming beam $p$ and the
outgoing hadron $a$. We we further generalize it to the azimuthal angle
dependent case as shown in the right panel of Fig. 1,

$\displaystyle\mathrm{{{EEC}}_{DIS}}(\tau,\phi)$
$\displaystyle=\frac{1}{2}\sum_{a}\int
d\theta_{ap}d\phi_{ap}dz_{a}z_{a}\frac{1}{\sigma}\frac{d\sigma}{dPS_{\rm
DIS}}\delta\left(\tau-\left(\frac{1+\cos\theta_{ap}}{2}\right)\right)\delta(\phi-\phi_{ap})\,.$
(9)

Here $dPS_{\rm DIS}=d\theta_{ap}d\phi_{ap}dz_{a}$. Similarly, in the back-to-
back limit ($\tau\to 0$), we find
$\displaystyle\tau={\bm{P}_{aT}^{2}}/({z_{a}^{2}Q^{2}})$, where
$Q=\sqrt{-q^{2}}$ is the invariant mass of the exchanged virtual photon,
$\bm{P}_{aT}$ is the transverse momentum of the outgoing hadron measured with
respect to the photon-beam axis. By introducing
$\bm{q}_{T}\equiv-\bm{P}_{aT}/z_{a}$, $\mathrm{{{EEC}}_{DIS}}(\tau,\phi)$ can
be related to the $\bm{q}_{T}$-differential cross-section in the back-to-back
limit as

$\displaystyle\mathrm{{{EEC}}}_{\rm
DIS}(\tau,\phi)=\frac{1}{\sigma}\frac{d\Sigma_{\rm DIS}}{d\tau d\phi}$
$\displaystyle=\frac{1}{2}\sum_{a}\int
d^{2}\bm{q}_{T}dz_{a}z_{a}\frac{1}{\sigma}\frac{d\sigma}{d^{2}\bm{q}_{T}dz_{a}}\delta\left(\tau-\frac{\bm{q}_{T}^{2}}{Q^{2}}\right)\delta(\phi-\phi_{q_{T}})\,.$
(10)

Starting from the relevant azimuthal angle dependent $\bm{q}_{T}$
factorization for the DIS process given in Gourdin:1972kro ; Kotzinian:1994dv
; Diehl:2005pc ; Bacchetta:2006tn ; Anselmino:2008sga ; Collins:2011zzd ;
Kang:2015msa , using the EEC jet functions defined above in Eqs. (3) and (4),
the EEC for SIDIS can be written as

$\displaystyle\frac{d\Sigma_{\rm DIS}}{dxdyd\tau d\phi}=$
$\displaystyle\frac{2\pi\alpha_{\rm em}^{2}}{Q^{2}}\frac{1+(1-y)^{2}}{y}\int
d^{2}{\bm{q}}_{T}\delta\left(\tau-\frac{{\bm{q}}_{T}^{2}}{Q^{2}}\right)\delta(\phi-\phi_{q_{T}})\int\frac{db\
b}{2\pi}\bigg{\\{}\mathcal{F}_{UU}$
$\displaystyle+\cos(2\phi_{q_{T}})\frac{2(1-y)}{1+(1-y)^{2}}\mathcal{F}_{UU}^{\cos(2\phi_{q_{T}})}+|{\bm{S}}_{\perp}|\sin(\phi_{q_{T}}-\phi_{s})\mathcal{F}_{UT}^{\sin(\phi_{q_{T}}-\phi_{s})}+\cdots\bigg{\\}}\,,$
(11)

where $\mathcal{F}_{UU}\sim f_{1}\otimes J_{q},\
\mathcal{F}_{UU}^{\cos\left(2\phi_{q_{T}}\right)}\sim h_{1}\otimes
J_{q}^{\perp},\ \mathcal{F}_{UT}^{\sin\left(\phi_{q_{T}}-\phi_{s}\right)}\sim
f_{1T}^{\perp}\otimes J_{q}$, etc. For the polarizations, $|{\bm{S}}_{\perp}|$
is the transverse spin of the incoming proton, $\lambda_{e}$ is the helicity
of the incoming electron, and the indices $A$ and $B$ of $\mathcal{F}_{AB}$
represent the polarization of the incoming electron and proton, respectively.
When studying a subset $\mathbb{S}$ of the produced hadrons, one has the
structure function given in the form of $\mathcal{F}_{AB}^{\mathbb{S}}$, with
an extra superscript $\mathbb{S}$. Additionally, the angles $\phi_{s}$ and
$\phi_{q_{T}}$ are the azimuthal angles of the ${\bm{S}}_{\perp}$ and
$\bm{q}_{T}$. For phenomenology, we plot the Collins asymmetry
$\mathcal{A}_{\rm DIS}^{\mathbb{S}}$ (left panel) and the Sivers asymmetry
$\mathcal{A}_{\rm DIS}^{\rm Sivers}$ (right panel) in Fig. 3, where

$\displaystyle\mathcal{A}_{\rm
DIS}^{\mathbb{S}}=\frac{\mathcal{F}_{UU}^{\mathbb{S},\cos\left(2\phi_{q_{T}}\right)}}{\mathcal{F}_{UU}^{\mathbb{S}}}\,,\qquad\mathcal{A}_{\rm
DIS}^{\rm
Sivers}=\frac{\mathcal{F}_{UT}^{\sin\left(\phi_{q_{T}}-\phi_{s}\right)}}{\mathcal{F}_{UU}}\,.$
(12)

For the Collins asymmetry, we use unpolarized TMD PDFs $f_{1}$, transversity
$h_{1}$ and Collins function $H_{1}^{\perp}$ fitted in Kang:2015msa . Although
the subset with $\mathbb{S}=\\{\pi^{+},\ \pi^{-}\\}$ is suppressed due to the
sum rule, one can have sizable asymmetries when choosing a subset of either
positively or negatively charged pions in EEC. In the right panel of Fig. 3,
we plot the Sivers asymmetry with all the final pions summed over. Here the
Sivers functions we adopted were extracted in Echevarria:2020hpy . The
asymmetry reaches a few percents at small $\tau$ limit, indicating this
observable as a promising tool for further study of the Sivers function.

Figure 3: $A_{\rm DIS}^{\mathbb{S}=\\{\pi^{\pm}\\},\ \\{\pi^{+},\ \pi^{-}\\}}$
(left) and $A_{\rm DIS}^{\rm Sivers}$ for $\mathbb{S}=\\{\pi^{+},\ \pi^{-}\\}$
(right) at EIC kinematics.

##  4 Conclusion

In this work, we generalize the conventional EEC observable to include the
azimuthal angle dependence, where a new EEC jet function referred to as the
Collins-type jet function arises. We study the application of this azimuthal
angle dependent EEC in both $e^{+}e^{-}$ and semi-inclusive deep inelastic
scattering, and find that they allow us to explore 3D structure of the
nucleon, especially the spin-dependent ones. Thus, the azimuthal angle
dependent EEC provides a novel direction for studying the 3D structure of
nucleons.

## References

  * (1) C. L. Basham, L. S. Brown, S. D. Ellis and S. T. Love, _Energy Correlations in electron - Positron Annihilation: Testing QCD_ , _Phys. Rev. Lett._ 41 (1978) 1585.
  * (2) C. L. Basham, L. S. Brown, S. D. Ellis and S. T. Love, _Energy Correlations in electron-Positron Annihilation in Quantum Chromodynamics: Asymptotically Free Perturbation Theory_ , _Phys. Rev. D_ 19 (1979) 2018.
  * (3) T. Kinoshita, _Mass singularities of Feynman amplitudes_ , _J. Math. Phys._ 3 (1962) 650–677.
  * (4) T. D. Lee and M. Nauenberg, _Degenerate Systems and Mass Singularities_ , _Phys. Rev._ 133 (1964) B1549–B1562.
  * (5) CELLO collaboration, H. J. Behrend et al., _Analysis of the Energy Weighted Angular Correlations in Hadronic $e^{+}e^{-}$ Annihilations at 22-GeV and 34-GeV_, _Z. Phys. C_ 14 (1982) 95.
  * (6) JADE collaboration, W. Bartel et al., _Measurements of Energy Correlations in $e^{+}e^{-}\to$ Hadrons_, _Z. Phys. C_ 25 (1984) 231.
  * (7) E. Fernandez et al., _A Measurement of Energy-energy Correlations in $e^{+}e^{-}\to$ Hadrons at $\sqrt{s}=29$-GeV_, _Phys. Rev. D_ 31 (1985) 2724.
  * (8) PLUTO collaboration, C. Berger et al., _A Study of Energy-energy Correlations in $e^{+}e^{-}$ Annihilations at $\sqrt{s}=34$.6-GeV_, _Z. Phys. C_ 28 (1985) 365.
  * (9) TASSO collaboration, W. Braunschweig et al., _A Study of Energy-energy Correlations Between 12-GeV and 46.8-GeV CM Energies_ , _Z. Phys. C_ 36 (1987) 349–361.
  * (10) D. R. Wood et al., _Determination of $\alpha^{-}s$ From Energy-energy Correlations in $e^{+}e^{-}$ Annihilation at 29-GeV_, _Phys. Rev. D_ 37 (1988) 3091.
  * (11) I. Moult and H. X. Zhu, _Simplicity from Recoil: The Three-Loop Soft Function and Factorization for the Energy-Energy Correlation_ , _JHEP_ 08 (2018) 160, [1801.02627].
  * (12) K. Gottfried and J. D. Jackson, _On the Connection between production mechanism and decay of resonances at high-energies_ , _Nuovo Cim._ 33 (1964) 309–330.
  * (13) Z.-B. Kang, K. Lee, D. Y. Shao and F. Zhao, _to appear , Collins-type Energy-Energy Correlators in the Back-to-Back Region_, 2023.
  * (14) Z.-B. Kang, A. Prokudin, P. Sun and F. Yuan, _Extraction of Quark Transversity Distribution and Collins Fragmentation Functions with QCD Evolution_ , _Phys. Rev. D_ 93 (2016) 014009, [1505.05589].
  * (15) H. T. Li, Y. Makris and I. Vitev, _Energy-energy correlators in Deep Inelastic Scattering_ , _Phys. Rev. D_ 103 (2021) 094005, [2102.05669].
  * (16) M. Gourdin, _Semiinclusive reactions induced by leptons_ , _Nucl. Phys. B_ 49 (1972) 501–512.
  * (17) A. Kotzinian, _New quark distributions and semiinclusive electroproduction on the polarized nucleons_ , _Nucl. Phys. B_ 441 (1995) 234–248, [hep-ph/9412283].
  * (18) M. Diehl and S. Sapeta, _On the analysis of lepton scattering on longitudinally or transversely polarized protons_ , _Eur. Phys. J. C_ 41 (2005) 515–533, [hep-ph/0503023].
  * (19) A. Bacchetta, M. Diehl, K. Goeke, A. Metz, P. J. Mulders and M. Schlegel, _Semi-inclusive deep inelastic scattering at small transverse momentum_ , _JHEP_ 02 (2007) 093, [hep-ph/0611265].
  * (20) M. Anselmino, M. Boglione, U. D’Alesio, A. Kotzinian, S. Melis, F. Murgia et al., _Sivers Effect for Pion and Kaon Production in Semi-Inclusive Deep Inelastic Scattering_ , _Eur. Phys. J. A_ 39 (2009) 89–100, [0805.2677].
  * (21) J. Collins, _Foundations of perturbative QCD_ , vol. 32.  Cambridge University Press, 11, 2013.
  * (22) M. G. Echevarria, Z.-B. Kang and J. Terry, _Global analysis of the Sivers functions at NLO+NNLL in QCD_ , _JHEP_ 01 (2021) 126, [2009.10710].