# Can Machines Resonate with Humans?
Evaluating the Emotional and Empathic Comprehension of LMs

Muhammad Arslan Manzoor1 Yuxia Wang1∗ Minghan Wang2 Preslav Nakov1
1MBZUAI, Abu Dhabi, UAE 2University of Monash, Melbourne, Australia
{muhammad.arslan, yuxia.wang<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Empathy plays a pivotal role in fostering prosocial behavior, often triggered
by the sharing of personal experiences through narratives. However, modeling
empathy using NLP approaches remains challenging due to its deep
interconnection with human interaction dynamics. Previous approaches, which
involve fine-tuning language models (LMs) on human-annotated empathic
datasets, have had limited success. In our pursuit of improving empathy
understanding in LMs, we propose several strategies, including contrastive
learning with masked LMs and supervised fine-tuning with Large Language Models
(LLMs). While these methods show improvements over previous methods, the
overall results remain unsatisfactory. To better understand this trend, we
performed an analysis which reveals a low agreement among annotators. This
lack of consensus hinders training and highlights the subjective nature of the
task. We also explore the cultural impact on annotations. To study this, we
meticulously collected story pairs in Urdu language and find that subjectivity
in interpreting empathy among annotators appears to be independent of cultural
background. The insights from our systematic exploration of LMs’ understanding
of empathy suggest that there is considerable room for exploration in both
task formulation and modeling.

##  1 Introduction

With Large Language Models (LLMs) demonstrating impressive capabilities in
generating naturally sounding answers over a broad range of human inquiries,
more individuals turn to seek solutions and emotional comfort by interacting
with LLM-supported chatbots OpenAI (2023); Chang et al. (2024). They express
thoughts, feelings and share their experiences, expecting deep understanding
and sympathetic responses from a chatbot that can resonate with them, as shown
in Figure 1 (Berridge et al., 2023). This requests LLMs to first fully
understand the event, the emotions, and the empathy in the narratives, and
then to respond appropriately Lin et al. (2023).

Figure 1: An ideal interaction between users and a system. A chatbot can
resonate with a human, and a search engine can retrieve stories of similar
experience.

Shen et al. (2023) proposed the task of measuring empathic similarity. It
assesses the similarity between two narratives describing personal experiences
from four aspects including event, emotion, moral, and empathy by a numerical
score ranging from 1 to 4. However, both fine-tuned LMs (BERT, RoBERTa) and
few-shot prompting LLMs achieved low correlation with human annotations. Our
error analysis shows that models can recognize dissimilar story pairs with
scores in range of 1-2, but struggle to distinguish fine-grained differences
between similar story pairs in range of 2.5-4 (see Section 2). This suggests
that the nuanced patterns and relationships in similar pairs are not captured
by current methods Wang et al. (2022).

We hypothesize that the LMs used in these methods were primarily trained for
semantic understanding tasks, rather than for emotion and empathy. They may
capture the similarity between events, but they struggle to understand the
complex social and emotional signals in human narratives Reimers and Gurevych
(2019), this is further discussed in Appendix A. Moreover, naïve LLM prompts
may not fully empower LLMs’ reasoning ability to make correct judgements. To
this end, we explore multiple strategies to improve the accuracy of empathic
similarity, including contrastive learning of LMs, LLM reasoning and fine-
tuning with and without Chain-of-Thought (CoT) (Wei et al., 2022). We achieve
around 5–10% improvement in the Pearson and Spearman correlation. However,
accuracy appears to be capped and cannot be improved beyond a certain value.

This motivated us to speculate that gold labels might be problematic due to
the subjective nature of judging emotion and empathy similarity. To
investigate this, we randomly sampled ten story pairs from EmpathicStories and
asked eight annotators with different background to annotate the similarity in
terms of event, emotion, moral and empathy. Overall low human agreement for
all aspects was observed, especially moral, followed by empathy, emotion; the
highest agreement was for the event. Interestingly, annotators from the same
country/culture, particularly who are friends, had much higher agreement than
others. We further collected a new Urdu dataset to explore the cultural impact
on empathy similarity labeling, which revealed similar findings.

In sum, this work presents three contributions:

  * •

We explore various approaches to improve empathic similarity estimation,
including the use of contrastive loss in LMs, reasoning and fine-tuning of
LLMs.

  * •

We analyze the upper bound of correlation between model predictions and human
gold labels. By gathering collective human opinions and measuring inter-
annotator agreement, we reveal high disagreements in empathic labeling,
highlighting the subjective nature of judging empathic similarity.

  * •

We collect a new Urdu dataset to investigate the impact of language and
culture on empathic labeling. Our analysis concludes that subjectivity in the
labeling is independent of the cultural background of annotators.

##  2 Dataset and Error Analysis

In this section, we first introduce the dataset used in this work and then we
perform an error analysis for some baseline methods.

###  2.1 Task and Dataset

In this work, we focus on the task of measuring the empathic similarity
between two narratives by a numerical score in the range of 1 to 4, with 1
representing totally dissimilar and 4 indicating extremely similar. We used
the EmpathicStories dataset, which was created by Shen et al. (2023). It
consists of 1,500 unique stories and 2,000 story pairs, split into 1,500 pairs
for training, 100 for development, and 400 for testing. Table 1 shows the
distribution of train, development, and test sets across different similarity
ranges. Each story has two versions, full and summary, in which the annotators
assign labels based on the summary from four perspectives including event,
emotion, moral, and overall empathy. Figure 2 shows the correlation between
the four similarity scores. We can see that the moral similarity has the
highest correlation with empathy, followed by event and emotion.

##### Evaluation Measurements

Pearson correlation ($r$) and Spearman rank correlation ($\rho$) are used to
measure the performance of systems for predicting emotional similarity. This
measure the linear correlation between the model outputs and the human
annotations, the degree of monotonicity under ranking, respectively. Mean
square error (MSE) assesses the models’ ability to get close to the gold
standard. In a discrete setting, following Shen et al. (2023), both the
predicted scores and the gold labels are binned into two classes thresholding
by 2.5 — label = 1 if score > 2.5; otherwise, it is 0. Then accuracy,
precision, recall and macro-F1 are used.

###  2.2 Error Analysis

Baseline: we reproduced the results of Shen et al. (2023) by fine-tuning SBERT
Reimers and Gurevych (2019) and BART-base Lewis et al. (2019) as Table 9,
given that prompting davinci-text-003 and GPT-3.5-turbo had lower accuracy. We
obtained Spearman correlation $\rho$ that was similar to that in the original
paper, with lower F1.

Splits$\rightarrow$ | Train | Dev. | Test. | SBERT | BART
---|---|---|---|---|---
Label $\downarrow$ | ($1500$) | ($100$) | ($400$) | #Dev | #Test | #Dev | #Test
1 | 85 | 2 | 15 | 0 | 0 | 0 | 1
1.5 | 125 | 16 | 29 | 0 | 0 | 2 | 3
2 | 310 | 24 | 75 | 1 | 1 | 11 | 19
2.5 | 288 | 20 | 76 | 4 | 17 | 15 | 33
3 | 344 | 21 | 103 | 13 | 49 | 11 | 55
3.5 | 262 | 15 | 84 | 12 | 70 | 10 | 57
4 | 86 | 2 | 18 | 2 | 18 | 2 | 15

Table 1: EmpathicStories dataset distribution and the number of incorrect
predictions on the development / test set by fine-tuned SBERT and BART
baselines.

##### Results and Analysis

We regard a predicted score as a severely incorrect estimation when the
absolute difference between the predicted score and the gold labels is greater
than 1.0,111The cosine similarity scores based on SBERT and BART is in the
range of [0,1], we scale them up by multiplying 4 to match the annotation
range of 1-4. the number of incorrect predictions over the development and the
test sets is shown in Table 1.

We found that the model excels at identifying dissimilar story pairs, but
struggles with similar pairs with nuanced differences. Specifically, the model
exhibits lower error rates in the score range of 1–2, and higher error rates
in the range of 2.5–4. Particularly for 4, both models have a 100% error rate
on the test set. We could attribute this to exposure bias during training.
However, the number of training examples scoring between 2.5 and 4 is actually
much larger than the number of dissimilar examples. This suggests that it is
difficult for models to discern the subtle differences between similar story
pairs, underlining a critical area for model improvement.

##### Hypothesis

We hypothesize that SBERT and BART are primarily pre-trained and fine-tuned to
learn semantic features, in which emotional and empathetic features are under-
represented. Despite the importance of understanding the semantics of the
stories, this does not necessarily account for capturing the deep, empathetic
connections between narratives, especially when these connections are subtler
and more implicit than straightforward lexical or thematic commonalities. The
failure to identify the connections between similar narratives in terms of
empathy leads to poor performance on similar story pairs ranging from 2.5 to
4, tending to be recognized as dissimilar with a score between 1-2. To
alleviate these issues and to enhance the model performance on empathetic
similarity task, we propose a variety of strategies in Section 3.

##  3 Enhancing Empathy Representation

In order to improve the model’s ability to recognize the connections between
narratives that are semantically diverse yet empathically similar, we apply
constrative losses in LM fine-tuning, and incorporate reasoning in LLM
inference and fine-tuning.

##### Contrastive Learning of LMs

We use a contrastive loss to enhance representation by bringing the embeddings
of similar examples closer, while pushing such of dissimilar samples apart in
the hidden space (Gao et al., 2021). This involves first grouping them into
positive and negative pairs, and then applying a contrastive loss to learn
patterns. In this task, we use annotated similarity scores to determine
positive and negative pairs, setting a threshold of 2.5 as the boundary
following the binning approach in Shen et al. (2023).

We explored various contrastive losses, including the ContrastiveLoss (Hadsell
et al., 2006), AnglELoss (Li and Li, 2024) and CoSENTLoss222www.sbert.net. We
also re-implemented the approach using cosine similarity loss as a baseline.
We used masked language models (MLMs) including MPNet (Song et al., 2020),
RoBERTa (base, large) (Liu et al., 2019) as text encoders leveraging their
bidirectional encoding capabilities and computational efficiency.

##### LLM Reasoning and Fine-tuning

We explored the potential of LLMs for the task of empathic similarity in both
inference and fine-tuning, with two prompting strategies:

  * •

Score-Only: with annotation guidelines, the LLM is prompted to predict only a
score without any additional content.

  * •

Chain-of-Thought (CoT): the LLM is instructed to first provide an explanation
and then to predict the scores.

For the CoT explanation, even though the human-annotated explanations were
provided in EmpathicStories, they were collected based on the story summary by
GPT-3.5-Turbo, which can be biased and inaccurate. Moreover, the distribution
of human-written text is distinct from the distribution of LLM-generated
reasons, which makes the reasons provided by the humans less useful for
lowering the perplexity in LLM fine-tuning (Liu et al., 2024a). To address
this, we prompted a powerful LLM to generate explanations guided by gold
labels, leveraging LLM capability of causal reasoning. That is, to explain why
two narratives have the similarity of [scores placeholder] from the aspects of
event, emotion, moral, and empathy.

##### Gold Label-Guided Explanation

Based on the pair of full stories with the ground truth similarity scores, we
asked the LLM to analyze the story pair from dimensions such as theme,
emotional content, characters, narrative structures, and overall empathy. The
generated analysis served as the reasoning content in supervised fine-tuning
(SFT). We explored both full parameter and PEFT Hu et al. (2021) fine-tuning.

##  4 Experiments

We experimented with discriminative LMs and generative LLMs with zero-shot and
fine-tuning.

Train Label$\downarrow$ | Model | Loss | $r$ | $\rho$ | F1-macro
---|---|---|---|---|---
EmpathicStories Baselines (Summary)
Empathy | SBERT | MSE | 0.359 | 0.352 | 0.713
Empathy | BART | MSE | 0.342 | 0.344 | 0.701
Empathy | GPT-3.5-turbo | NA/5-shot | 0.278 | 0.281 | 0.696
Contrastive Loss of LMs (Summary)
Empathy | RoBerta-base | Cosine | 0.404 | 0.388 | 0.608
Empathy | Multi-qa-MPNet | Cosine | 0.400 | 0.395 | 0.647
Event | RoBerta-base | Contrastive | 0.318 | 0.309 | 0.634
Event | Multi-qa-MPNet | Contrastive | 0.370 | 0.364 | 0.624
Emotion | RoBerta-base | Cosine | 0.377 | 0.371 | 0.650
Emotion | RoBerta-large | Cosine | 0.393 | 0.388 | 0.611
Moral | RoBerta-base | AnglELoss | 0.326 | 0.323 | 0.649
Moral | Multi-qa-MPNet | Cosine | 0.387 | 0.374 | 0.604

Table 2: Test set results for LM fine-tuned over annotations of event,
emotion, moral, and overall empathy score and tested on empathy vs. baseline
results obtained from Shen et al. (2023).

Testbed$\rightarrow$ | Summary | Full
---|---|---
Test Label$\downarrow$ | $r$ | $\rho$ | MSE$\downarrow$ | Acc | Prec | Recall | F1-macro | $r$ | $\rho$ | MSE$\downarrow$ | Acc | Prec | Recall | F1-macro
OpenAI-text-embedding-3-large
Empathy | 0.336 | 0.329 | 1.510 | 0.505 | 0.633 | 0.517 | 0.376 | 0.362 | 0.363 | 1.440 | 0.507 | 0.624 | 0.519 | 0.384
Event | 0.485 | 0.465 | 0.620 | 0.780 | 0.738 | 0.542 | 0.522 | 0.488 | 0.469 | 0.590 | 0.782 | 0.737 | 0.551 | 0.538
Emotion | 0.392 | 0.388 | 1.310 | 0.550 | 0.582 | 0.510 | 0.392 | 0.393 | 0.386 | 1.260 | 0.568 | 0.685 | 0.529 | 0.421
Moral | 0.366 | 0.356 | 1.210 | 0.620 | 0.692 | 0.525 | 0.437 | 0.395 | 0.403 | 1.140 | 0.618 | 0.651 | 0.524 | 0.440
Llama-3-8B
Empathy | 0.325 | 0.322 | 0.620 | 0.595 | 0.596 | 0.593 | 0.591 | 0.324 | 0.308 | 0.520 | 0.590 | 0.595 | 0.592 | 0.588
Event | 0.315 | 0.306 | 1.240 | 0.525 | 0.574 | 0.601 | 0.509 | 0.342 | 0.312 | 0.900 | 0.660 | 0.617 | 0.659 | 0.611
Emotion | 0.270 | 0.265 | 0.780 | 0.555 | 0.564 | 0.563 | 0.554 | 0.317 | 0.294 | 0.600 | 0.595 | 0.590 | 0.588 | 0.588
Moral | 0.319 | 0.323 | 0.830 | 0.600 | 0.622 | 0.623 | 0.600 | 0.331 | 0.329 | 0.640 | 0.650 | 0.636 | 0.638 | 0.637
Llama-3-70B
Empathy | 0.405 | 0.403 | 0.620 | 0.635 | 0.661 | 0.630 | 0.614 | 0.304 | 0.295 | 0.970 | 0.545 | 0.628 | 0.534 | 0.436
Event | 0.427 | 0.431 | 1.280 | 0.480 | 0.623 | 0.639 | 0.479 | 0.337 | 0.357 | 1.980 | 0.305 | 0.625 | 0.547 | 0.287
Emotion | 0.387 | 0.374 | 0.770 | 0.565 | 0.605 | 0.585 | 0.550 | 0.305 | 0.312 | 1.180 | 0.495 | 0.617 | 0.532 | 0.407
Moral | 0.412 | 0.415 | 0.840 | 0.585 | 0.663 | 0.636 | 0.579 | 0.305 | 0.320 | 1.320 | 0.455 | 0.659 | 0.545 | 0.391
GPT-4o
Empathy | 0.442 | 0.441 | 0.620 | 0.652 | 0.660 | 0.655 | 0.650 | 0.350 | 0.373 | 0.650 | 0.640 | 0.640 | 0.640 | 0.640
Event | 0.492 | 0.488 | 0.560 | 0.703 | 0.660 | 0.716 | 0.659 | 0.414 | 0.424 | 0.710 | 0.605 | 0.615 | 0.660 | 0.579
Emotion | 0.466 | 0.452 | 0.580 | 0.647 | 0.645 | 0.641 | 0.641 | 0.360 | 0.371 | 0.660 | 0.620 | 0.622 | 0.622 | 0.620
Moral | 0.476 | 0.481 | 0.560 | 0.698 | 0.685 | 0.687 | 0.686 | 0.396 | 0.424 | 0.630 | 0.685 | 0.689 | 0.697 | 0.683

Table 3:  Zero-shot results of discriminative and generative models on the
test set using summary and full story over four types of gold similarity
scores: empathy, event, emotion and moral. Cosine similarity is scaled 1-4 by
$\times$4 for discriminative models. Classification gold labels are binned by
$\text{score}>2.5$.

###  4.1 Discriminative LMs

We refer to discriminative models as small pre-trained LMs such as RoBERTa,
and sentence embedding models like SBERT. For both the zero-shot and the fine-
tuning settings, we calculated the cosine similarity based on embeddings of
two stories derived from pre-trained or fine-tuned sentence encoders.

Each story pair in EmpathicStories has four similarity scores: event, emotion,
moral, and overall empathy. Shen et al. (2023) only focused on fine-tuning and
evaluating over empathy scores, under-investigating the other three aspects
and their impact on the empathy similarity estimations. In contrast, we
experimented across all four labels, providing a comprehensive understanding
of the model performance.

####  4.1.1 Zero-Shot Evaluation

We performed these experiments on both the full stories and on the summaries.
We calculated the cosine similarity based on the embeddings of a range of pre-
trained text encoders including open-sourced MiniLM (Wang et al., 2020) and
MPNet (Song et al., 2020), as well as close-sourced OpenAI-text-
embedding-3-large (Neelakantan et al., 2022), and then we evaluated their
predictions against the gold labels from four perspectives. Table 10 shows
that (_i_) the close-sourced model outperformed open-sourced models on both
full and summary across all aspects in correlations, and (_ii_) the cosine
similarity scores for all models had the highest correlation with the event
labels, indicating that semantics was the dominant feature captured in story
representations. They were significantly notable than emotional and moral
signals.

####  4.1.2 Fine-Tuning with Contrastive Loss

We examined the effectiveness of contrastive losses presented in Section 3
based on three LMs. We fine-tuned them on the four types of gold labels
separately and evaluated them using the overall empathy score as ground-truth
based on pairs of story summary.333The full story always exceeds the maximum
input length of these three LMs, resulting in poor performance. This allows us
to discern the influence of each attribute on the empathy similarity modeling.

Table 2 showcases the best two results over multiple LM$\times$Loss settings
given the training label. We found that training on empathy similarity yields
the highest correlation, followed by emotion, moral, and event across all
settings. This suggests that empathy is more closely related to emotion and
moral than to events. In terms of loss functions, CosineSimilarity loss
consistently outperformed the rest, except for event as training labels, where
Contrastive loss was the best. This highlights the robustness of
CosineSimilarity loss across different scores. Among the three LMs, Roberta-
base was always in the top-2 list, demonstrating robustness over all settings.
Roberta-large and Multi-qa-MPNet outperformed in correlations when training
with event, emotion, and moral.

These results suggest that, while some models and loss functions excel in
specific metrics or training labels, Roberta-base with CosineSimilarity loss
appears to be the most robust and effective combination for capturing the
nuances of empathy. See the full results in Table 11.

###  4.2 Generative LLMs

Unlike discriminative models to predict or measure a similarity score, LLMs
explicitly generate scores in natural language. We first evaluated LLMs in
zero-shot setting to assess the inherent ability of LLMs to comprehend emotion
and empathy, and then we fine-tuned the same LLMs to learn and to reason with
the empathy in the stories.

####  4.2.1 Zero-Shot Evaluation

We optimized the prompt used in Shen et al. (2023) by specifying the
differences between the four concepts, and by highlighting this in the system
prompt instead of the user input, guiding the model to predict empathic
similarity scores. Observing that the order of the stories affected the
predicted scores, we generated two scores for each pair of stories by swapping
the positions of stories A and B and then averaging the two scores as the
final prediction.

Table 3 shows two major findings: (_i_) Zero-shot LLMs have clear advantages
over cosine similarity based on sentence embeddings. $r$=0.442 for GPT-4o vs.
$r$=0.336 for OpenAI-text-embedding on story summary. The larger the model,
the higher the correlations with the gold empathy scores. This suggests that
LLMs benefit from large-scale pre-training that empowers the
distinguishability of nuanced empathic differences in narratives. (_ii_)!Over
all models and over the four types of scores, using the summaries is superior
to using the full stories, which aligns with the fact that the gold labels are
annotated based on the summary.

Testbed$\rightarrow$ | Development Set | Test Set
---|---|---
SFT Strategy | $r$ | $\rho$ | MSE | Acc | Prec | Recall | F1-macro | $r$ | $\rho$ | MSE | Acc | Prec | Recall | F1-macro
LoRA-SFT-Score | 0.470 | 0.476 | 0.497 | 0.790 | 0.759 | 0.786 | 0.768 | 0.307 | 0.320 | 0.643 | 0.600 | 0.602 | 0.603 | 0.599
LoRA-SFT-COT | 0.342 | 0.325 | 0.618 | 0.690 | 0.698 | 0.709 | 0.687 | 0.289 | 0.299 | 0.651 | 0.613 | 0.619 | 0.609 | 0.603
Full-SFT-Score | 0.209 | 0.189 | 0.627 | 0.630 | 0.640 | 0.632 | 0.625 | 0.028 | 0.039 | 0.773 | 0.507 | 0.506 | 0.506 | 0.504

Table 4: LLaMA-3-8B SFT by three paradigms. LoRA-SFT-Score: score-only
strategy tuned with LoRA. LoRA-SFT-COT: CoT strategy with LoRA. Full-SFT-Score
tunes the model with full parameters on empathy scores.

####  4.2.2 Is SFT Helpful?

To further investigate the potential of LLMs in this task, we performed
supervised fine-tuning (SFT) with LLaMA-3-8B using the story summaries and the
corresponding empathic similarity scores from the training set. In our
experiments, we explored two different prompting strategies as introduced in
Section 3. For the Score-Only strategy, we implemented both full-parameter and
LoRA (Hu et al., 2021) SFT. For the CoT strategy, we only performed LoRA SFT
due to the memory pressure of the longer sequence length. For all strategies,
we tuned the model for two epochs.

Table 4, shows that, compared to the zero-shot setting, SFT does not enhance
the performance of LLaMA-3-8B regardless of the strategy. In some cases, fine-
tuning even worsens the results. Moreover, the full-parameter SFT setting
yields the poorest results, which is counter-intuitive. This motivated us to
analyze why LLMs could not improve the performance of empathic similarity to
$r$>0.5 at least, and why SFT even worsened the results.

Gold$\rightarrow$ | y=1 | y=2 | y=3 | y=4
---|---|---|---|---
$P(Y|X_{y_{gold}=1})$ | 0.142 | 0.392 | 0.409 | 0.058
$P(Y|X_{y_{gold}=2})$ | 0.140 | 0.386 | 0.418 | 0.056
$P(Y|X_{y_{gold}=3})$ | 0.129 | 0.381 | 0.434 | 0.055
$P(Y|X_{y_{gold}=4})$ | 0.132 | 0.378 | 0.428 | 0.062
$P(Y)$ | 0.140 | 0.399 | 0.404 | 0.057

Table 5: Probability estimated by fine-tuned LLaMA-3-8B over four groups
binned by gold label y of story pairs. $X_{y_{gold}=1}$ is the group of
examples whose gold empathy similarity is 1. $P(Y)$ represents empirical
distribution of training labels. All values are computed on the training set.

###  4.3 Understanding the Bottleneck

In order to understand what LLMs learned in the fine-tuning and why they
struggle with this task, we conducted an analysis of the predictions made by
the full-parameter fine-tuned model. We chose the full-parameter fine-tuned
model as it has been adapted to the task of empathy similarity estimation,
which reduces the interference from its prior knowledge.

Ideally, if the model learned how to predict the empathic similarity based on
the input story pairs, its probability over four similarity classes [1,2,3,4]
should be close to [1, 0, 0, 0] when the gold label of the input pair is 1,
and to [0, 1, 0, 0] when the gold label of the input pair is 2, and so forth.
To observe the real predictive probability over each similarity class, we
first grouped the training story pairs $X$ into four sets by their gold
empathy labels $y$ (continuous scores are rounded to the nearest integer),
denoted as $X_{y_{gold}=i}$ with $i\in[1,2,3,4]$. Next, we computed the
probability for each pair by applying the softmax function to the logits of
$\\{1,2,3,4\\}$ of the first predicted token, and then we averaged the
probabilities across all pairs within each class, as shown in Table 5.

Regardless of the similarity class, the predicted probability kept the same as
the empirical distribution of training data over four classes, i.e., $P(Y)$ =
(0.140, 0.399, 0.404, 0.057), which is calculated by counting the percentage
of story pairs falling into each class. That is, after fine-tuning, the model
learned nothing about how to map the input $x\rightarrow y$, but merely the
distribution of $P(Y)$. During inference, the model could not estimate the
corresponding score conditioned on the input story pair, but sampled a
similarity class based on the distribution of $P(Y)$ whatever the input pair
was, leading to randomness on the development and the test sets (see Figure
4).

Based on these observations and previous findings in semantic textual
similarity that models would be confused when training with ambiguous and
subjective labels (Wang et al., 2023, 2024), we speculate that the annotated
empathic similarity scores are substantially subjective, i.e., that labels of
high human disagreements hinder the model to learn distinguishable patterns.
This could also explain why LMs and LLMs could not improve the performance by
a large margin despite our extensive efforts. The struggle of the models to
overcome the bottleneck can be attributed to the complexity and the
variability in the human interpretations of these abstract concepts. This
highlights the necessity to critically analyze the quality of the dataset and
to address the inherent subjectivity in its labels.

##  5 Collective Human Opinions

To analyze the subjectivity in empathic similarity labeling, we first
collected eight annotations for each English story pairs, and then we
investigated the impact of the language/culture on the empathy labeling by
collecting and annotating Urdu story pairs.

IAA Metric$\downarrow$ | S_eve | S_emo | S_mor | S_emp | F_eve | F_emo | F_mor | F_emp | Comb_eve | Comb_emo | S_eve | S_emo | S_mor | S_emp | F_eve | F_emo | F_mor | F_emp | Comb_eve | Comb_emo
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
All annotators - 8 | English / Chinese Speakers (friends- 2) |
Pearson | 0.421 | 0.078 | 0.189 | 0.191 | 0.442 | 0.098 | 0.110 | -0.015 | 0.176 | 0.141 | 0.735 | 0.322 | 0.192 | 0.148 | 0.848 | 0.311 | -0.086 | -0.241 | -0.302 | -0.153
Spearman | 0.395 | 0.063 | 0.187 | 0.190 | 0.438 | 0.088 | 0.138 | 0.004 | 0.158 | 0.129 | 0.701 | 0.252 | 0.143 | 0.090 | 0.868 | 0.313 | -0.109 | -0.108 | -0.437 | -0.183
KA | 0.349 | 0.059 | 0.168 | 0.114 | 0.398 | 0.092 | 0.105 | 0.027 | 0.095 | 0.107 | 0.401 | 0.086 | 0.174 | -0.152 | 0.761 | 0.233 | -0.065 | -0.349 | -0.295 | -0.128
Cohen Kappa | 0.124 | -0.015 | 0.067 | 0.047 | 0.157 | -0.024 | 0.030 | 0.026 | 0.059 | 0.049 | 0.143 | -0.045 | 0.024 | -0.013 | 0.359 | -0.053 | 0.067 | -0.039 | -0.364 | -0.061
English / Urdu Speakers (colleagues - 4) | English / Urdu speakers (sisters - 2) |
Pearson | 0.266 | 0.037 | 0.089 | 0.093 | 0.322 | -0.017 | 0.234 | -0.039 | 0.137 | 0.071 | 0.836 | 0.728 | 0.698 | 0.771 | 0.683 | 0.697 | 0.278 | 0.504 | 0.625 | 0.238
Spearman | 0.235 | 0.028 | 0.101 | 0.078 | 0.295 | -0.013 | 0.194 | -0.075 | 0.133 | 0.065 | 0.754 | 0.686 | 0.695 | 0.787 | 0.523 | 0.577 | 0.301 | 0.455 | 0.577 | 0.221
KA | 0.235 | 0.051 | 0.050 | 0.026 | 0.311 | -0.008 | 0.187 | 0.057 | 0.098 | 0.030 | 0.724 | 0.476 | 0.709 | 0.773 | 0.659 | 0.560 | 0.273 | 0.486 | 0.317 | 0.195
Cohen Kappa | 0.113 | 0.053 | -0.030 | 0.007 | 0.114 | -0.040 | -0.006 | -0.070 | 0.015 | -0.016 | 0.429 | 0.167 | 0.375 | 0.412 | 0.412 | -0.045 | 0.143 | 0.403 | 0.138 | -0.014

Table 6: Agreement scores for four groups. The summary and the full story are
abbreviated as S and F, Comb = S+F. Event, emotion, moral, and empathy are
shortened as eve, emo, mor, and emp. KA refers to Krippendorff’s Alpha.

###  5.1 English Story Pair Annotation

We sampled 10 story pairs from the development sets of EmpathicStories, and we
invited eight annotators to assign labels in three settings.

##### Annotation Setup

Shen et al. (2023) employed Amazon MTurk workers to annotate similarity scores
based on the story summaries considering the heavy cognition load of
understanding the full story. Upon performing qualitative analysis of both
full stories and their summaries, we observed that summaries generated by
GPT-3.5-turbo would dismiss many narratives presenting emotional changes (from
depressed, to sad, finally turn to happy), inner monologue (I feel alone after
Mary left) and details about other roles, only keeping the major events and
the tone of the full stories. Yet, these details are very important to
identify subtle feelings and they can affect empathy. Model predictions based
on the full stories also significantly differ from such obtained from the
summaries as shown in Section 4. To this end, we collected annotations under
three settings:

  * •

Continuous score on the summary: the annotators assign similarity scores from
event, emotion, moral, and empathy as continuous value ranging from 1 to 4,
based on the story summary.

  * •

Continuous score on the full story: same as above, but based on the full
stories.

  * •

Discrete class-label on full story: considering that continuous scores provide
larger labeling space, which may exacerbate subjectivity and inner-annotator
disagreement, the annotators are asked to rate event and emotion similarity
using class labels: very similar (V), moderately similar (M), and not similar
(N). Here, we did not rate moral and empathy given they are too abstract
concept to perceive and rate.

Based on the annotation results, we aim to measure and to analyze the
subjectivity of the human opinions in different annotation setups, i.e., how
the representation of the stories and the annotation scale influence inter-
annotator agreement (IAA). Pearson and Spearman correlation, Krippendorff’s
Alpha, and Cohen Kappa (convert to discrete) are used to measure IAA. We first
calculated the agreement between each pair of annotators and then we averaged
them.

##### Annotator Background

We had eight annotators, aged between 20 and 35 years, from diverse cultural
and ethnic backgrounds, including two native Chinese speakers and six native
Urdu speakers, with a balanced gender distribution. All annotators are
proficient in English, some majored in psychology with bachelor’s degree and
some are PhD and postdoc students in NLP. Training was performed before the
formal annotations, which instructed the annotators how to rate the similarity
score and highlighted the differences between event, emotion, moral, and
empathy (see the annotation guidelines in Figure 5).

##### Results and Analysis

Table 6 shows the inter-annotator agreements. We can see that the agreement
for event is the highest, followed by emotion, empathy, and moral. This
suggests that it is easier for the annotators to reach agreement on more
concrete aspects such as event and emotion compared to more abstract concepts
of moral and empathy.

The closer the relationship between two annotators, the higher the agreement
for more subtle aspects. Two Chinese friends (one male and one female)
exhibited the highest correlation on judging event similarity based on the
full story, but lower correlation on all other settings than the two sisters
from Pakistan. They even achieved $r/\rho$ >0.7 for moral and empathy when the
average was less than 0.2 based on story summary. Four Urdu speaker colleagues
showed extremely low agreement with each other, with correlations around 0.3
for event and $r/\rho$<0.1 for other aspects.

This indicates that individuals having similar background and experiences tend
to reach agreement in interpretations of the stories, particular for subtle
and abstract concepts. Moreover, varying the agreement scores across different
groups suggests that individual background and culture significantly influence
how stories are perceived and annotated. This subjectivity poses challenges
for models to learn and to replicate human judgements, especially for moral
and empathy. The agreement of collective human opinions in similarity score
annotations also presents an upper for a model.

Comparing the three settings, we can see lower agreement when labeling with
three classes than with continuous scores. Moreover, using full stories yields
higher correlations compared to using summaries. This guides the setting of
future annotations for empathic similarity, providing annotators with full
stories and annotating concrete aspects like event and emotion using
continuous labels.

Given that none of the annotators is an English native speaker or has grown in
a Western culture, bias might be introduced in labeling English stories pairs.
We thus collected Urdu stories and asked Urdu native speakers to annotate
them. This aims to eliminate potential biases, and most importantly to
investigate the impact of languages and culture on the empathy similarity
labeling.

###  5.2 Urdu Dataset Construction

Urdu is widely spoken in many South Asian countries, including Pakistan,
India, Bangladesh, and Nepal. Roman Urdu/Hindi, which uses the English
alphabet to write Urdu or Hindi, is commonly used in these regions for
communication, discussions, and sharing feelings on media platforms.
Consequently, we collected stories written in Roman Urdu to capture the
authentic expression and nuances of this widely used form of communication.

IAA Metric$\downarrow$ | event | emotion | empathy | overall
---|---|---|---|---
| All annotators (4) | GPT-4o
Pearson | 0.308 | 0.381 | 0.422 | 0.425
Spearman | 0.307 | 0.362 | 0.392 | 0.403
KA | 0.214 | 0.316 | 0.392 | 0.392
Cohen Kappa | 0.133 | 0.169 | 0.190 | 0.075

Table 7: Agreement scores for Urdu annotators on the Roman Urdu dataset. On
the right, the empathy scores of 4 annotators are averaged correspondingly and
compared to the overall GPT score.

Gold Label$\downarrow$ | $r$ | $\rho$ | MSE$\downarrow$ | Acc | Prec | Recall | F1
---|---|---|---|---|---|---|---
OpenAI-text-embedding-3-large
Empathy | 0.486 | 0.527 | 1.610 | 0.296 | 0.609 | 0.562 | 0.289
Event | 0.539 | 0.548 | 1.040 | 0.399 | 0.651 | 0.564 | 0.372
Emotion | 0.493 | 0.529 | 1.600 | 0.292 | 0.607 | 0.562 | 0.286
Llama-3-8B
Empathy | 0.410 | 0.375 | 0.640 | 0.759 | 0.772 | 0.629 | 0.637
Event | 0.445 | 0.371 | 0.450 | 0.832 | 0.733 | 0.651 | 0.675
Emotion | 0.428 | 0.412 | 0.480 | 0.808 | 0.680 | 0.615 | 0.632
GPT-4o
Empathy | 0.715 | 0.725 | 0.330 | 0.797 | 0.762 | 0.757 | 0.759
Event | 0.772 | 0.762 | 0.310 | 0.842 | 0.760 | 0.848 | 0.786
Emotion | 0.774 | 0.774 | 0.310 | 0.818 | 0.732 | 0.807 | 0.754

Table 8: Performance on the Urdu dataset over three types of gold similarity
scores: event, emotion, and empathy using cosine similarity ($v_{a}$, $v_{b}$)
with OpenAI embedding and zero-shot LLMs.

##### Synthetic Story Pairs Generation

After manually checking the quality of the stories generated by GPT-4o (e.g.,
whether they are emotion-rich stories matching local culture), we collected
300 story pairs by instructing GPT-4o to generate story pairs with varying
similarities given a diverse range of domains and topics (see prompts in
Figure 6). Each pair consisted of theme, content of two stories, an overall
similarity score s in [1,4], and a brief explaination of the score.

##### Human Annotations

Four Urdu native speakers were trained to annotate similarity scores for
event, emotion and empathy, ranging from 1 to 4 with increments of 0.5; we
excluded moral due to high ambiguity. The IAA between the four annotators is
shown in Table 7. The last column overall represents the agreement between the
averaged empathy scores of the four raters with the overall similarity score s
provided by GPT-4o during story generation. Interestingly, empathy achieved
the highest IAA, followed by emotion and then event, which differs with the
findings in English data: event>emotion>empathy. This confirms highly
subjective nature of the task. The correlations between the averaged empathy
score and GPT-4o similarity are closer to those between annotators, suggesting
the human-like judgement of GPT-4o.

##### LLM Estimations

Given the superior zero-shot performance obtained by OpenAI-text-
embedding-3-large, LLaMA-3-8B, and GPT-4o on English dataset, we applied them
on the Urdu story pairs in Table 8. GPT-4o outperforms the other two models by
a sizable margin. All of them have the highest scores on event, and then
emotion and empathy, which is similar to the results for English, while the
correlations of all aspects are much higher than for English across the three
models.

##  6 Conclusion and Future Work

We proposed a variety of strategies to enhance model performance on empathic
similarity, including the use of contrastive losses on LMs, LLM reasoning and
fine-tuning. Our experiments demonstrated 5–10% improvements compared to
baselines. However, our analysis revealed the subjective nature of empathic
similarity between narratives. Collective human annotations on both English
and Urdu story pairs illustrated the low human agreement in empathy labeling,
highlighting the inherent challenges of this task and indicating an upper
bound that a model can reach. The annotation of Urdu stories further exposed
the cultural impact on empathic labeling.

In future work, we will explore task reformulations to reduce the variability
and subjectivity, and we will try more robust approaches to effectively model
empathy in narratives.

## Limitation

The overall task inherently carries subjectivity. The gold labels are not
standardized and vary based on individual backgrounds, demographics,
perspectives, experiences, and surroundings. Cultural differences also play a
significant role. Emotions are complex and varied, and each emotion can be
expressed in multiple ways and at different intensities. Thus, gold labels are
subjective, and stories themselves contain many nuances that may not be fully
empathized by the annotators. Each emotion has a broad spectrum of intensity,
rather than a binary state like happy or sad.

## Ethical Statement

We adhere to ethical standards in data collection, annotation, and analysis.
All human annotators were well informed about the task and provided their
consent. We ensured diverse representation among the annotators to account for
various cultural and demographic perspectives, aiming to minimize biases in
empathic and emotional labeling. The datasets used, including those in Urdu,
were collected and processed with respect for cultural sensitivity. We
acknowledge the subjective nature of empathy and emotion analysis and have
taken steps to highlight and to address these challenges in our study. Our
work is committed to advancing understanding while maintaining ethical
integrity and respect for the individuals whose data and annotations were
used.

## References

  * Berridge et al. (2023) Clara Berridge, Yuanjin Zhou, Julie M. Robillard, and Jeffrey Kaye. 2023.  Companion robots to mitigate loneliness among older adults: Perceptions of benefit and possible deception.  _Frontiers in Psychology_ , 14.
  * Buecker et al. (2021) Susanne Buecker, Marcus Mund, Sandy Chwastek, Melina Sostmann, and Maike Luhmann. 2021.  Is loneliness in emerging adults increasing over time? a preregistered cross-temporal meta-analysis and systematic review. _Psychological bulletin_ , 147 8:787–805.
  * Chang et al. (2024) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_ , 15(3):1–45.
  * Chen et al. (2024) Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. 2024.  Tombench: Benchmarking theory of mind in large language models.  _CoRR_ , abs/2402.15052.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA_ , pages 1735–1742. IEEE Computer Society.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Huang et al. (2024) Junqin Huang, Zhongjie Hu, Zihao Jing, Mengya Gao, and Yichao Wu. 2024.  Piccolo2: General text embedding with multi-task hybrid loss training.  _arXiv preprint arXiv:2405.06932_.
  * Hubert et al. (2024) Nicolas Hubert, Pierre Monnin, Armelle Brun, and Davy Monticolo. 2024.  Treat different negatives differently: Enriching loss functions with domain and range constraints for link prediction.  In _European Semantic Web Conference_ , pages 22–40. Springer.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Li and Li (2024) Xianming Li and Jing Li. 2024.  Angle-optimized text embeddings.
  * Lin et al. (2023) Tzu-Mi Lin, Jung-Ying Chang, and Lung-Hao Lee. 2023.  NCUEE-NLP at WASSA 2023 shared task 1: Empathy and emotion prediction using sentiment-enhanced RoBERTa transformers.  In _Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis_, pages 548–552, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2024a) Qin Liu, Fei Wang, Nan Xu, Tianyi Yan, Tao Meng, and Muhao Chen. 2024a.  Monotonic paraphrasing improves generalization of language model prompting.  _CoRR_ , abs/2403.16038.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Liu et al. (2024b) Zhiwei Liu, Kailai Yang, Tianlin Zhang, Qianqian Xie, Zeping Yu, and Sophia Ananiadou. 2024b.  Emollms: A series of emotional large language models and annotation tools for comprehensive affective analysis.  _CoRR_ , abs/2401.08508.
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022.  Text and code embeddings by contrastive pre-training.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Sharma et al. (2021) Ashish Sharma, Inna Wanyin Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2021.  Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach.  _Proceedings of the Web Conference 2021_.
  * Sharma et al. (2022) Ashish Sharma, Inna Wanyin Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2022.  Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support.  _Nature Machine Intelligence_ , 5:46–57.
  * Sharma et al. (2020) Ashish Sharma, Adam S. Miner, David C. Atkins, and Tim Althoff. 2020.  A computational approach to understanding empathy expressed in text-based mental health support.  _ArXiv_ , abs/2009.08441.
  * Shen et al. (2023) Jocelyn Shen, Maarten Sap, Pedro Colon-Hernandez, Hae Park, and Cynthia Breazeal. 2023.  Modeling empathic similarity in personal narratives.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6237–6252, Singapore. Association for Computational Linguistics.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  _Advances in neural information processing systems_ , 33:16857–16867.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
  * Wang et al. (2022) Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. 2022.  Uncertainty estimation and reduction of pre-trained models for text regression.  _Transactions of the Association for Computational Linguistics_ , 10:680–696.
  * Wang et al. (2023) Yuxia Wang, Shimin Tao, Ning Xie, Hao Yang, Timothy Baldwin, and Karin Verspoor. 2023.  Collective human opinions in semantic textual similarity.  _Transactions of the Association for Computational Linguistics_ , 11:997–1013.
  * Wang et al. (2024) Yuxia Wang, Minghan Wang, and Preslav Nakov. 2024.  Rethinking STS and NLI in large language models.  In _Findings of the Association for Computational Linguistics: EACL 2024_ , pages 965–982, St. Julian’s, Malta. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * West et al. (2023) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian R. Fisher, Abhilasha Ravichander, Khyathi Raghavi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2023.  The generative ai paradox: "what it can create, it may not understand".  _ArXiv_ , abs/2311.00059.
  * Wu et al. (2021) Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021.  Esimcse: Enhanced sample building method for contrastive learning of unsupervised sentence embedding.  In _International Conference on Computational Linguistics_.
  * Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024.  Tree of thoughts: Deliberate problem solving with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_.

## Appendix

##  Appendix A Related Work

##### Emotion and Empathy in LMs

Many prior work focus on tasks of recognizing (mixed) emotions, scoring
intensity, inferring and explaining person’s emotional reactions (Liu et al.,
2024b; Chen et al., 2024). Sharma et al. (2020) collect EPITOME dataset to
facilitate understanding empathy levels in online mental health conversations.
Based on this dataset, Sharma et al. (2022) explore how human-AI collaboration
could enhance empathic communication in text-based mental health support. They
find that peer supporters can effectively use AI-generated feedback to improve
their responses to support seekers. Sharma et al. (2021) further automate this
process by leveraging a reinforcement learning agent, which makes sentence-
level edits on supporters’ posts to increase empathy while maintaining
conversation quality.

Recent study explored in general domains illustrates that current LLMs are far
better than human in generation but fall short of understanding (West et al.,
2023). However, for empathy-demanding scenarios, understanding and resonating
with support-seekers are even more important than outputting opinions Buecker
et al. (2021). In this work, we aim to examine language models capability in
understanding and identifying nuanced difference of emotion and empathy, based
on the task of empathic similarity (Shen et al., 2023). It examines models
understanding of human narratives in both emotion and empathy by assessing the
similarity between two stories from event, emotion, moral and empathy.
However, predictions show low correction with human-annotated labels by both
fine-tuning LMs (e.g., SBERT and BART) on empathic similarity dataset and
prompting GPT-3.5-turbo. We speculate that the sentence representations may
capture adequate semantic context, but lack emotional and empathic signals. In
this work, we aim to improve the estimation accuracy by forcing models to
discern and learn the underlying reasons for similarities between pairs of
stories, via (1) enhancing LM-based sentence embeddings, and (2) empowering
LLM reasoning capability.

##### Sentence Embedding Enhancement

Many previous methods improve semantic embeddings by adjusting loss functions
in training. SimCSE leverages contrastive loss Gao et al. (2021), and ESimCSE
applies momentum contrast strategy to increase the number of negative pairs
involved in the loss calculation, showing notable improvements across multiple
semantic textual similarity benchmarks (Wu et al., 2021). Hubert et al. (2024)
improved the knowledge graph embeddings by integrating semantic awareness into
traditional loss functions like pairwise hinge loss, binary cross-entropy
loss, and logistic loss. Huang et al. (2024) use a multi-task hybrid loss,
incorporating both skeletal and semantic embedding into loss functions for
micro-gesture classification. Therefore, we investigate a variety of loss
functions such as ContrastiveLoss, CosineSimilarityLoss, CoSENTLoss and
AnglELoss, based on LMs including MPNet Song et al. (2020), RoBERTa (base,
large) Liu et al. (2019), DeBERTa (small, large) He et al. (2020) to improve
sentence embedding in representing empathic features.

##### LLM Reasoning and Fine-tuning

Naive LLM prompts may fail to fully leverage LLMs’ reasoning capabilities,
leading to poor accuracy on EmpathicStories dataset Shen et al. (2023). Recent
advancements in zero- and few-shot prompting techniques may boost performance
such as Chain-of-Thought prompting (CoT) Wei et al. (2022), Least-to-Most
prompting Zhou et al. (2022), and search-based approaches like Tree-of-Thought
(ToT) Yao et al. (2024). We prompt LLMs by CoT and also fine-tune LLMs with
reasoning trajectories, encouraging LLMs to think over before making the final
decision.

##  Appendix B Error Analysis

Figure 2: Pearson and Spearsman correlation between overall empathic
similarity, event, emotion and moral similarity. Moral similarity has the
highest correlation with the empathic, followed by event and emotion.

Model | Macro-F1$\uparrow$ | MSE$\downarrow$ | $\rho\uparrow$
---|---|---|---
SBERT (our rerun) | 0.560 | 0.224 | 0.317
SBERT (theirs) | 0.712 | – | 0.352
BART (our rerun) | 0.579 | 0.240 | 0.389
BART (theirs) | 0.706 | – | 0.344

Table 9: Reproduced results vs. results presented in Shen et al. (2023).

Figure 3: Dev/Test set empathic similarity distribution: predictions of BART
vs. SBERT vs. ground truth.

##  Appendix C Experimental Results

###  C.1 Discriminative Models Results

Testbed$\rightarrow$ | Development Set | Test Set
---|---|---
Test Label$\downarrow$ | $r$ | $\rho$ | MSE$\downarrow$ | Acc | Prec | Recall | F1-macro | $r$ | $\rho$ | MSE$\downarrow$ | Acc | Prec | Recall | F1-macro
Multi-qa-MiniLM (Summary)
Empathy | 0.204 | 0.201 | 0.0815 | 0.63 | 0.591 | 0.543 | 0.513 | 0.220 | 0.213 | 0.095 | 0.550 | 0.590 | 0.557 | 0.508
Event | 0.331 | 0.263 | 0.061 | 0.84 | 0.711 | 0.670 | 0.686 | 0.311 | 0.294 | 0.056 | 0.740 | 0.623 | 0.609 | 0.614
Emotion | 0.271 | 0.231 | 0.0792 | 0.72 | 0.643 | 0.578 | 0.578 | 0.226 | 0.226 | 0.088 | 0.560 | 0.554 | 0.534 | 0.500
Moral | 0.169 | 0.133 | 0.075 | 0.760 | 0.600 | 0.568 | 0.574 | 0.216 | 0.214 | 0.084 | 0.610 | 0.575 | 0.550 | 0.533
Multi-qa-MiniLM (Full)
Empathy | 0.226 | 0.236 | 0.078 | 0.600 | 0.520 | 0.509 | 0.466 | 0.151 | 0.13 | 0.092 | 0.520 | 0.543 | 0.527 | 0.475
Event | 0.337 | 0.285 | 0.060 | 0.790 | 0.593 | 0.569 | 0.577 | 0.268 | 0.239 | 0.058 | 0.715 | 0.584 | 0.574 | 0.578
Emotion | 0.206 | 0.208 | 0.082 | 0.690 | 0.572 | 0.537 | 0.524 | 0.226 | 0.226 | 0.088 | 0.560 | 0.554 | 0.534 | 0.500
Moral | 0.169 | 0.139 | 0.075 | 0.760 | 0.600 | 0.568 | 0.574 | 0.216 | 0.214 | 0.084 | 0.610 | 0.575 | 0.550 | 0.533
Multi-qa-MPNet-base (Summary)
Empathy | 0.170 | 0.140 | 0.066 | 0.440 | 0.554 | 0.528 | 0.405 | 0.263 | 0.238 | 0.058 | 0.540 | 0.581 | 0.530 | 0.447
Event | 0.298 | 0.280 | 0.133 | 0.270 | 0.515 | 0.513 | 0.269 | 0.368 | 0.347 | 0.102 | 0.315 | 0.575 | 0.538 | 0.303
Emotion | 0.224 | 0.204 | 0.086 | 0.370 | 0.544 | 0.525 | 0.355 | 0.318 | 0.297 | 0.067 | 0.515 | 0.632 | 0.549 | 0.445
Moral | 0.206 | 0.198 | 0.093 | 0.310 | 0.539 | 0.528 | 0.306 | 0.274 | 0.256 | 0.075 | 0.445 | 0.571 | 0.527 | 0.392
Multi-qa-MPNet-base (Full)
Empathy | 0.145 | 0.127 | 0.076 | 0.440 | 0.6380 | 0.543 | 0.384 | 0.299 | 0.288 | 0.065 | 0.530 | 0.645 | 0.518 | 0.389
Event | 0.291 | 0.303 | 0.158 | 0.230 | 0.524 | 0.512 | 0.223 | 0.349 | 0.346 | 0.132 | 0.250 | 0.500 | 0.500 | 0.218
Emotion | 0.218 | 0.228 | 0.100 | 0.350 | 0.589 | 0.532 | 0.320 | 0.350 | 0.333 | 0.078 | 0.465 | 0.537 | 0.504 | 0.347
Moral | 0.231 | 0.227 | 0.108 | 0.290 | 0.614 | 0.550 | 0.277 | 0.301 | 0.282 | 0.089 | 0.420 | 0.625 | 0.516 | 0.330
OpenAI-text-embedding-3-large (Summary)
Empathy | 0.335 | 0.315 | 1.280 | 0.630 | 0.813 | 0.513 | 0.411 | 0.336 | 0.329 | 1.510 | 0.505 | 0.633 | 0.517 | 0.376
Event | 0.437 | 0.411 | 0.600 | 0.820 | 0.414 | 0.494 | 0.451 | 0.485 | 0.465 | 0.620 | 0.780 | 0.738 | 0.542 | 0.522
Emotion | 0.394 | 0.350 | 1.130 | 0.720 | 0.859 | 0.517 | 0.451 | 0.392 | 0.388 | 1.310 | 0.550 | 0.582 | 0.510 | 0.392
Moral | 0.359 | 0.309 | 0.960 | 0.800 | 0.899 | 0.524 | 0.489 | 0.366 | 0.356 | 1.210 | 0.620 | 0.692 | 0.525 | 0.437
OpenAI-text-embedding-3-large (Full)
Empathy | 0.303 | 0.309 | 1.400 | 0.620 | 0.561 | 0.505 | 0.406 | 0.362 | 0.363 | 1.440 | 0.507 | 0.624 | 0.519 | 0.384
Event | 0.385 | 0.409 | 0.680 | 0.810 | 0.413 | 0.488 | 0.448 | 0.488 | 0.469 | 0.590 | 0.782 | 0.737 | 0.551 | 0.538
Emotion | 0.345 | 0.361 | 1.260 | 0.710 | 0.607 | 0.510 | 0.446 | 0.393 | 0.386 | 1.260 | 0.568 | 0.685 | 0.529 | 0.421
Moral | 0.337 | 0.325 | 1.050 | 0.790 | 0.648 | 0.517 | 0.484 | 0.395 | 0.403 | 1.140 | 0.618 | 0.651 | 0.524 | 0.440

Table 10: Cosine similarity($v_{a}$, $v_{b}$) across three sentence embedding
models. Results on dev and test sets over four type of gold similarity scores:
empathy, event, emotion and moral based on full story and summary. Cosine
similarity is normalized to the scale 1-4 by $\times$4\. Classification gold
labels are binned by $\text{score}>2.5$.

Model | Loss | $r$ | $\rho$ | Acc | Prec | Recall | F1-macro | $r$ | $\rho$ | Acc | Prec | Recall | F1-macro
---|---|---|---|---|---|---|---|---|---|---|---|---|---
| Empathy | Event
RoBerta-base | AnglELoss | 0.355 | 0.351 | 0.640 | 0.642 | 0.641 | 0.640 | 0.254 | 0.248 | 0.593 | 0.608 | 0.597 | 0.583
RoBerta-base | Cosine | 0.404 | 0.389 | 0.620 | 0.629 | 0.616 | 0.609 | 0.337 | 0.315 | 0.565 | 0.622 | 0.573 | 0.520
RoBerta-base | Contrastive | 0.331 | 0.319 | 0.637 | 0.639 | 0.636 | 0.634 | 0.318 | 0.309 | 0.635 | 0.637 | 0.636 | 0.635
RoBerta-large | ConSENT | 0.291 | 0.298 | 0.603 | 0.611 | 0.605 | 0.599 | 0.281 | 0.299 | 0.590 | 0.602 | 0.594 | 0.583
RoBerta-large | Cosine | 0.373 | 0.346 | 0.608 | 0.612 | 0.604 | 0.599 | 0.353 | 0.350 | 0.593 | 0.653 | 0.600 | 0.557
Multi-qa-MPNet | Cosine | 0.400 | 0.396 | 0.647 | 0.648 | 0.648 | 0.647 | 0.343 | 0.346 | 0.573 | 0.643 | 0.581 | 0.525
Multi-qa-MPNet | Contrastive | 0.358 | 0.347 | 0.615 | 0.615 | 0.614 | 0.613 | 0.371 | 0.365 | 0.625 | 0.629 | 0.627 | 0.624
| Emotion | Moral
RoBerta-base | AnglE | 0.292 | 0.279 | 0.580 | 0.581 | 0.581 | 0.580 | 0.326 | 0.324 | 0.650 | 0.650 | 0.650 | 0.650
RoBerta-base | Cosine | 0.378 | 0.371 | 0.652 | 0.654 | 0.651 | 0.650 | 0.339 | 0.334 | 0.645 | 0.645 | 0.645 | 0.645
RoBerta-base | Contrastive | 0.335 | 0.321 | 0.618 | 0.626 | 0.614 | 0.607 | 0.346 | 0.331 | 0.642 | 0.645 | 0.644 | 0.642
RoBerta-large | CoSENT | 0.303 | 0.287 | 0.585 | 0.585 | 0.585 | 0.585 | 0.287 | 0.311 | 0.650 | 0.651 | 0.651 | 0.650
RoBerta-large | Cosine | 0.394 | 0.388 | 0.620 | 0.626 | 0.617 | 0.611 | 0.387 | 0.373 | 0.640 | 0.640 | 0.640 | 0.640
Multi-qa-MPNet | Cosine | 0.367 | 0.359 | 0.613 | 0.612 | 0.611 | 0.611 | 0.387 | 0.374 | 0.608 | 0.615 | 0.610 | 0.605
Multi-qa-MPNet | Contrastive | 0.339 | 0.325 | 0.610 | 0.610 | 0.609 | 0.608 | 0.368 | 0.362 | 0.608 | 0.616 | 0.610 | 0.604

Table 11: Performance of LMs fine-tuned based on annotations of event, emotion
and moral & overall empathy similarity scores respectively. Note that we
consistently evaluate against empathy similarity score (test gold labels)
though training with labels from four aspects.

###  C.2 Generative LLMs Results

Testbed$\rightarrow$ | Development Set | Test Set
---|---|---
Test Label$\downarrow$ | $r$ | $\rho$ | MSE | Acc | Prec | Recall | F1-macro | $r$ | $\rho$ | MSE | Acc | Prec | Recall | F1-macro
Llama-3-8B (Summary)
Empathy | 0.494 | 0.500 | 0.550 | 0.740 | 0.751 | 0.765 | 0.738 | 0.325 | 0.322 | 0.620 | 0.595 | 0.596 | 0.593 | 0.591
Event | 0.468 | 0.465 | 1.340 | 0.570 | 0.597 | 0.671 | 0.531 | 0.315 | 0.306 | 1.240 | 0.525 | 0.574 | 0.601 | 0.509
Emotion | 0.459 | 0.466 | 0.780 | 0.630 | 0.648 | 0.678 | 0.619 | 0.270 | 0.265 | 0.780 | 0.555 | 0.564 | 0.563 | 0.554
Moral | 0.411 | 0.400 | 0.900 | 0.590 | 0.614 | 0.671 | 0.563 | 0.319 | 0.323 | 0.830 | 0.600 | 0.622 | 0.623 | 0.600
Llama-3-8B (Full)
Empathy | 0.326 | 0.351 | 0.540 | 0.670 | 0.646 | 0.637 | 0.640 | 0.324 | 0.308 | 0.520 | 0.590 | 0.595 | 0.592 | 0.588
Event | 0.208 | 0.212 | 1.220 | 0.680 | 0.577 | 0.620 | 0.573 | 0.342 | 0.312 | 0.900 | 0.660 | 0.617 | 0.659 | 0.611
Emotion | 0.235 | 0.233 | 0.780 | 0.660 | 0.600 | 0.608 | 0.603 | 0.317 | 0.294 | 0.600 | 0.595 | 0.590 | 0.588 | 0.588
Moral | 0.165 | 0.147 | 0.860 | 0.600 | 0.502 | 0.502 | 0.493 | 0.331 | 0.329 | 0.640 | 0.650 | 0.636 | 0.638 | 0.637
Llama-3-70B (Summary)
Empathy | 0.450 | 0.459 | 0.610 | 0.700 | 0.705 | 0.717 | 0.697 | 0.405 | 0.403 | 0.620 | 0.635 | 0.661 | 0.630 | 0.614
Event | 0.420 | 0.403 | 1.400 | 0.570 | 0.583 | 0.647 | 0.524 | 0.427 | 0.431 | 1.280 | 0.480 | 0.623 | 0.639 | 0.479
Emotion | 0.416 | 0.435 | 0.840 | 0.630 | 0.639 | 0.668 | 0.616 | 0.387 | 0.374 | 0.770 | 0.565 | 0.605 | 0.585 | 0.550
Moral | 0.369 | 0.369 | 0.960 | 0.610 | 0.622 | 0.683 | 0.579 | 0.412 | 0.415 | 0.840 | 0.585 | 0.663 | 0.636 | 0.579
Llama-3-70B (Full)
Empathy | 0.264 | 0.284 | 1.110 | 0.460 | 0.621 | 0.554 | 0.421 | 0.304 | 0.295 | 0.970 | 0.545 | 0.628 | 0.534 | 0.436
Event | 0.380 | 0.402 | 2.220 | 0.270 | 0.549 | 0.537 | 0.268 | 0.337 | 0.357 | 1.980 | 0.305 | 0.625 | 0.547 | 0.287
Emotion | 0.344 | 0.333 | 1.380 | 0.390 | 0.617 | 0.560 | 0.372 | 0.305 | 0.312 | 1.180 | 0.495 | 0.617 | 0.532 | 0.407
Moral | 0.288 | 0.262 | 1.580 | 0.330 | 0.619 | 0.576 | 0.325 | 0.305 | 0.320 | 1.320 | 0.455 | 0.659 | 0.545 | 0.391
GPT-4o (Summary)
Empathy | 0.482 | 0.479 | 0.620 | 0.740 | 0.726 | 0.709 | 0.714 | 0.442 | 0.441 | 0.620 | 0.652 | 0.660 | 0.655 | 0.650
Event | 0.474 | 0.457 | 0.660 | 0.710 | 0.605 | 0.662 | 0.608 | 0.492 | 0.488 | 0.560 | 0.703 | 0.660 | 0.716 | 0.659
Emotion | 0.518 | 0.514 | 0.590 | 0.750 | 0.700 | 0.712 | 0.705 | 0.466 | 0.452 | 0.580 | 0.647 | 0.645 | 0.641 | 0.641
Moral | 0.423 | 0.416 | 0.650 | 0.710 | 0.621 | 0.659 | 0.628 | 0.476 | 0.481 | 0.560 | 0.698 | 0.685 | 0.687 | 0.686
GPT-4o (Full)
Empathy | 0.351 | 0.351 | 0.630 | 0.710 | 0.697 | 0.705 | 0.699 | 0.350 | 0.373 | 0.650 | 0.640 | 0.640 | 0.640 | 0.640
Event | 0.385 | 0.367 | 0.850 | 0.660 | 0.616 | 0.702 | 0.595 | 0.414 | 0.424 | 0.710 | 0.605 | 0.615 | 0.660 | 0.579
Emotion | 0.397 | 0.385 | 0.670 | 0.680 | 0.654 | 0.683 | 0.653 | 0.360 | 0.371 | 0.660 | 0.620 | 0.622 | 0.622 | 0.620
Moral | 0.271 | 0.236 | 0.780 | 0.660 | 0.622 | 0.680 | 0.609 | 0.396 | 0.424 | 0.630 | 0.685 | 0.689 | 0.697 | 0.683

Table 12: Results of Generative LLMs under zero-shot configuration using the
full and summary version story. Classification gold labels are binned by
$\text{score}>2.5$.

Figure 4: Confusion matrix of the first digit prediction.

##  Appendix D Annotation Guidelines

This figure shows the guidelines provided to annotators for obtaining
annotations on the Roman Urdu dataset. Annotators were first familiarized with
established tasks in the English language before being introduced to the Urdu
dataset, which is their native language.

Figure 5: Annotation guidelines provided to annotators.

##### Recruitment And Payment

Four Urdu native speakers were employed to annotate Urdu dataset. We pay 1 AED
per story pair for each annotator, in total of 300 * 4 = 1200 AED.

##  Appendix E Urdu Story Pair Generation

Figure 6 displays the prompt we prepared to synthetically generate the Roman
Urdu story data. To maintain consistency between the English and Urdu
datasets, we incorporated all the themes mentioned in Shen et al. (2023).
Figure 7 shows one of the 300 story pairs produced by GPT-4o.

Figure 6: Prompt used for Urdu story pair generation based on GPT-4o. Figure
7: Example of Urdu story pair.

##  Appendix F English Story Pair Example

This figure shows a sample pair of stories along with their summarized
versions and scores in four aspects: empathy, event, emotion, and moral.

Figure 8: Example of English story pair.


# Comparing and improving hybrid deep learning algorithms for identifying and
locating primary vertices

S Akar1 0000-0003-0288-9694 M Peters1 H Schreiner2 M D Sokoloff1
0000-0001-6181-4583 W Tepe1 1 University of Cincinnati, Cincinnati, OH 45221,
USA  2 Princeton University, Princeton, NJ 08544, USA<EMAIL_ADDRESS>
###### Abstract

Using deep neural networks to identify and locate proton-proton collision
points, or primary vertices, in LHCb has been studied for several years.
Preliminary results demonstrated the ability for a hybrid deep learning
algorithm to achieve similar or better physics performances compared to
standard heuristic approaches. The previously studied architectures relied
directly on hand-calculated Kernel Density Estimators (KDEs) as input
features. Calculating these KDEs was slow, making use of the DNN inference
engines in the experiment’s real-time analysis (trigger) system problematic.
Here we present recent results from a high-performance hybrid deep learning
algorithm that uses track parameters as input features rather than KDEs,
opening the path to deployment in the real-time trigger system.

##  1 Introduction

The LHCb experiment was recently upgraded for Run 3 of the LHC. It will record
proton-proton collision data at five times the instantaneous luminosity of Run
2. The average number of visible primary vertices (PVs), proton-proton
collisions in the detector closest to the beam-crossing region, will increase
from 1.1 to 5.6. The experiment will move to a pure software data ingestion
and trigger system, eliminating the Level 0 hardware trigger altogether [1]. A
conventional PV finding algorithm [2, 3] that satisfies all requirements
defined in the Trigger Technical Design Report [4] serves as the baseline. In
parallel, we have been developing a hybrid machine learning algorithm,
designed to run in the initial stage of the LHCb upgrade trigger.

The initial algorithm defined a one-dimensional Kernel Density Estimator (KDE)
histogram, plus two more one-dimensional histograms, to describe the
probabilities of tracks traversing small voxels in space. These feature sets
were modified to use two KDEs rather than one. [5, 6, 7]. “Classic”
convolutional Neural Networks (CNNs), referred to as KDE-to-hists models,
produce one-dimensional histograms that nominally predicts Gaussian peaks at
the locations of true PVs using three (or four) input histograms as their
feature sets. A hand-written clustering algorithm identifies the candidate PVs
and their positions. A first set of results used a “toy Monte Carlo” with
proto-tracking [5]. Using track parameters produced by the LHCb Run 3 Vertex
Locator (VELO) tracking algorithm [8] leads to significantly better
performance [6].

The original KDE [6] is a projection of a three-dimensional probability
distribution in voxels that has contributions only when two tracks pass close
to each other. Calculating this KDE exactly is very time-consuming. One of the
goals of our project is to predict PV positions directly from tracks’
parameters. A proof-of-concept was presented last year [7]. We trained a
model, tracks-to-KDE, predicting the KDE using the tracks parameters as input
and merged it with a trained version of a KDE-to-hists model. Mediocre
performance, compared to the original KDE-to-hists model, was achieved after a
final training allowing all weights of the merged tracks-to-hists model to
float.

In the last PV-finder update [6], we also presented studies of alternative
KDE-to-hists model architectures. In particular, we showed that the capacity
of our original CNN architecture [5] to learn increased with the number of
parameters, as expected from first principles. The study of a modified version
of the popular U-Net architecture [9], showed similar performances compared to
the best classic CNN architectures, and it trained more quickly. Here we
present improved results from an updated architecture for tracks-to-hists. A
brief description of the model in given in Sec. 2. Performance is discussed in
Sec. 3. A summary is presented in Sec. 4

##  2 Model architecture and training strategy

Figure 1:  This diagram illustrates the end-to-end, tracks-to-hist, deep
neural network used to predict an event’s target histogram from its tracks’
poca-ellipsoids. Each event is now sliced in 40 independent 100-bin
histograms. Six fully connected layers populate 8 100-bin channels in the last
of these layers, for each track. These contributions are summed and processed
by a U-Net model with 5 convolutional layers to construct the final 100-bin
histogram.

Building on our previous experience, we define a merged tracks-to-hists model;
this architecture is based on the one used in the proof-of-concept presented
in the last update [7]. The latest tracks-to-hists model, whose architecture
is shown in Fig. 1, includes a few updates: the tarcks-to-KDE part of the
model consists of 6 fully connected layers that are initially trained to
produce a KDE and the weights of the first 5 layers are temporarily frozen; a
variation with 8 latent feature sets is merged to the trained KDE-to-hists DNN
where the classical CNN layers are replaced by a U-Net model. Critically, we
also updated the structure of the input data for training and inference. In
the original approach, the one-dimensional feature sets consisted of 4000 bins
along the z-direction (beamline), each $100\,{\rm\mu m}$ wide, spanning the
active area of the VELO around the interaction point, such that
$z\in[-100,300]\,{\rm mm}$. In place of representing each event by a single
4000-bin histogram (feature set), we now slice each event into 40 slices of
100 bins each. This approach is motivated by the fact that the shapes of the
target histogram are expected to be invariant as a function of the true PV
position and it is easier for a DNN to learn to predict target histograms over
a smaller range of bins. Also, with an average of $\sim 5$ PVs per event, most
of the bins in both the KDE and target histograms have no significant
activity. The 40 slices of 100 bins are independent and homogeneous between
events. Each slice is treated independently, after which the predicted
4000-bin histogram is stitched back together.

Training on nVidia RTX 2080Ti GPUs is performed in several steps. First,
weights for the tracks-to-KDE model that uses individual tracks parameters
evaluated at their points of closest approach (poca-ellipsoid) to the beamline
[7] as input features to predict the KDE are determined. Then, the U-Net KDE-
to-hists model is trained using the hand-calculated KDEs as the input feature
set. Finally all weights and biases in the combined tracks-to-hists model are
allowed to float. An asymmetry parameter between the cost of overestimating
contributions to the target histograms and underestimating them [5] is used as
a hyperparameter to allow higher efficiency by incurring higher false positive
rates.

Figure 2:  Plots illustrating typical examples of target (shown in red) versus
predicted (shown in blue) histogram. Each of the four plots is drawn from
different slices of different individual events. Each of the 100 bins in the
histogram corresponds to a range in the $z$-direction of $100\,\mu$m.
Predicted PVs that are matched to the correct true PV (majority of cases) are
identified by a green tick in the plots. Example of missed true PV (orange
cross label) and false positive prediction (FP label) are shown in the bottom-
left and bottom-right plots, respectively.

##  3 Performances

### Evaluation

Performance evaluation is obtained by a matching procedure done by a heuristic
algorithm, based on the PV positions along the beam axis, $z$. Figure 2 shows
typical examples of predicted and target histograms using the final tracks-to-
hists model. In the previous report of PV-finder performances [5, 6, 7] a
predicted PV was matched if the distance, $\Delta z$, between it’s position,
$z_{\rm pred}$, and the true PV position, $z_{\rm true}$, satisfied $\Delta
z=|z_{\rm pred}-z_{\rm true}|\leq 0.5\,{\rm mm}$. The false positive rate is
obtained from the ratio of remaining predicted PVs after the matching
procedure over the total number of true PVs. This approach suffered a few
limitations, independent of the intrinsic DNN algorithm performances. For
instance, it is known that a PVs resolution depends on the number of tracks
originating from it. In case of low-multiplicity PVs, the predicted histogram
from the DNN algorithm, similar in shape to the target histogram, can be
displaced more than $0.5\,{\rm mm}$ from the true PV position. This scenario
can simultaneously “produce” a missed PV (reduced efficiency) and a false
positive signal (increased false positive rate). To address this issue, a new
matching procedure is used. Instead of a fixed window around the predicted PV
position, a resolution function is introduced in the matching procedure. The
resolution for a given predicted PV, $\sigma(z)$, is a function of the
standard deviation of the predicted histogram multiplied by a constant
parameter that can be tuned. Using this definition, a predicted PV is now
matched if $\Delta z=|z_{\rm pred}-z_{\rm true}|\leq 5\sigma(z)$. In Fig. 2 we
observe that for matched PVs, the standard deviation of the predicted peaks
(blue entries) is slightly larger than that of the corresponding true PVs (red
entries), estimated from the number of tracks. Peaks with larger standard
deviations are typically associated with PVs with fewer tracks.

### Predicted position and efficiency

The reported results come from statistically independent validation samples.
Using the matching procedure described above, we quantified the performances
of the trained tracks-to-hists DNN on a sample of 10K events corresponding to
slightly more than 50K true PVs. Figure 3 shows the efficiency of finding true
PVs as a function of the number of fully reconstructed originating tracks,
labelled LHCb long tracks, defined as tracks with hits in all the elements of
the LHCb tracking system. We observe that the efficiency is very high
($>99\%$) and stable for PVs with more than 20 tracks, and rapidly decreasing
for PVs with fewer tracks. Also shown in Fig. 3 is the distribution of the
distance, $\Delta z$, between the predicted and true PV position for matched
PVs. From a fit using a Gaussian distribution, we observe a small bias of
$-16.1\,\mathrm{\mu m}$ on the predicted PV position. Most notably, the
majority of matched PVs have a $\Delta z$ value below the bin width of the
target histogram of $0.1\,\mathrm{mm}$.

Figure 3:  (left) Efficiency of finding true PVs as a function of the number
of fully reconstructed originating tracks. The integrated efficiency and false
positive rate are also reported. (right) Distribution of the distance, $\Delta
z$, between the predicted and true PV position for matched PVs as defined in
the text, overlaid by a fitted Gaussian.  Figure 4:  Comparison between the
performances of models reported in previous years and the newest model
(magenta diamonds). An asymmetry parameter described in the text is varied to
produce the families of points observed.

### Performances evolution

Figure 4 shows how the performance of the DNN algorithms have evolved over
time. The efficiency is shown on the horizontal axis and the false positive
rate per event is shown on the vertical axis. The solid blue circles show the
performance of any early KDE-to-hists model described at ACAT-2019 [5]. The
green squares show the performances of a KDE-to-hists described at Connecting-
the-Dots in 2020 [6]. Both of the above models were trained using “toy Monte
Carlo” with proto-tracking. All subsequent DNN models were trained using full
VELO tracking algorithm [8], leading to significantly better performances (red
triangles to be compared to green squares). The cyan circles and the yellow
squares correspond to the best achieved performances for KDE-to-hists models
using either a classical CNN architecture or a U-Net model described at
CHEP-2021 [7]. The performances of all above models were obtained using the
“old” matching procedure with a fixed searching window of $0.5\,{\rm mm}$. The
magenta diamonds show the performance of the tracks-to-hist model described in
Sec. 2. These performances are obtained using the improved matching procedure
described above. The performance of the new tracks-to-hist model enables the
DNN to simultaneously reach high efficiencies ($>97\%$) and low false positive
rates ($0.03$ per event or $0.6\%$ per reconstructed PV).

##  4 Summary and Conclusions

Since we presented results at CHEP 2021, we have updated the tracks-to-hists
model to produce the first competitive end-to-end hybrid deep learning
algorithms for identifying and locating primary vertices based solely on
tracks’ parameters as input features. The performance of this model reaches
very high efficiencies with low false positive rates. By removing the (very
slow) KDE calculation used as part of the hybrid KDE-to-hists models, we have
opened the path to include a PV finding DNN in the real-time trigger system of
the experiment. Studies are currently ongoing to validate the feasibility of
including a version of the tracks-to-hists model to execute in the CUDA cores
(or in the tensor cores) of the GPUs constituting the LHCb trigger system. We
also plan on studying the effect of quantization on the model performances, as
well as the size of the model itself (number of nodes). Our specific
algorithms are being designed for use in LHCb with its Run 3 detector.
Developing and deploying machine learning inference engines that are highly
performant and satisfy computing system constraints requires sustained effort.
The results reported here should encourage work focused on using DNNs for
identifying vertices in other high energy physics experiments as well.

##  5 Acknowledgments

The authors thank the LHCb computing and simulation teams for their support
and for producing the simulated LHCb samples used in this paper. The authors
also thank the full LHCb Real Time Analysis team, especially the developers of
the VELO tracking algorithm [8] used to generate the “full LHCb MC” results
presented in Fig 4.

This work was supported, by the U.S. National Science Foundation under
Cooperative Agreement OAC-1836650 and awards PHY-1806260, and OAC-1450319.

## References

  * [1] Aaij R et al. (LHCb) 2014 LHCb Trigger and Online Upgrade Technical Design Report URL https://cds.cern.ch/record/1701361
  * [2] Reiss F 2020 Excerpts from the LHCb cookbook — RECEPTS for testing lepton universality and reconstructing primary vertices presented 18 12 2020 URL https://cds.cern.ch/record/2749592
  * [3] Reiss F et al. (LHCb) 2020 Fast parallel Primary Vertex reconstruction for the LHCb Upgrade talk presented at Connecting-the-Dots 2020 URL https://indico.cern.ch/event/831165/timetable/?view=standard
  * [4] Aaij R et al. (LHCb) 2019 JINST 14 P04013 (Preprint 1812.10790)
  * [5] Fang R, Schreiner H F, Sokoloff M D, Weisser C and Williams M 2020 J. Phys. Conf. Ser. 1525 012079 (Preprint 1906.08306)
  * [6] Akar S, Boettcher T J, Carl S, Schreiner H F, Sokoloff M D, Stahl M, Weisser C and Williams M 2020 An updated hybrid deep learning algorithm for identifying and locating primary vertices (Preprint 2007.01023)
  * [7] Akar S, Atluri G, Boettcher T, Peters M, Schreiner H, Sokoloff M, Stahl M, Tepe W, Weisser C and Williams M 2021 EPJ Web Conf. 251 04012 (Preprint 2103.04962)
  * [8] Hennequin A, Couturier B, Gligorov V, Ponce S, Quagliani R and Lacassagne L 2020 JINST 15 P06018 (Preprint 1912.09901)
  * [9] Ronneberger O, Fischer P and Brox T 2015 Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 ed Navab N, Hornegger J, Wells W M and Frangi A F (Cham: Springer International Publishing) pp 234–241 ISBN 978-3-319-24574-4 (Preprint 1505.04597)