# Emotional Theory of Mind: Bridging Fast Visual Processing with Slow
Linguistic Reasoning

Yasaman Etesam  Ozge Nilay Yalcin  Chuxuan Zhang  Angelica Lim
Simon Fraser University, BC, Canada
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
https://rosielab.github.io/Fast-and-Slow/

###### Abstract

The emotional theory of mind problem in images is an emotion recognition task,
specifically asking "How does the person in the bounding box feel?" Facial
expressions, body pose, contextual information and implicit commonsense
knowledge all contribute to the difficulty of the task, making this task
currently one of the hardest problems in affective computing. The goal of this
work is to evaluate the emotional commonsense knowledge embedded in recent
large vision language models (CLIP, LLaVA) and large language models (GPT-3.5)
on the Emotions in Context (EMOTIC) dataset. In order to evaluate a purely
text-based language model on images, we construct "narrative captions"
relevant to emotion perception, using a set of 872 physical social signal
descriptions related to 26 emotional categories, along with 224 labels for
emotionally salient environmental contexts, sourced from writer’s guides for
character expressions and settings. We evaluate the use of the resulting
captions in an image-to-language-to-emotion task. Experiments using zero-shot
vision-language models on EMOTIC show that combining "fast" and "slow"
reasoning is a promising way forward to improve emotion recognition systems.
Nevertheless, a gap remains in the zero-shot emotional theory of mind task
compared to prior work trained on the EMOTIC dataset.

Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic
Reasoning

Yasaman Etesam and Ozge Nilay Yalcin and Chuxuan Zhang and Angelica Lim Simon
Fraser University, BC, Canada<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>https://rosielab.github.io/Fast-and-Slow/

##  1 Introduction

Figure 1:  Fast frameworks might perceive the body pose with raised arms and
predict the emotions of ‘surprise’ and ‘fear’. Slow networks could further
reason about context and future implications.

Imagine a photo depicting a mother looking up from a swimming pool at her
4-year old daughter perched at the edge of a high diving board. How does this
person feel? Traditional computer vision approaches might detect the swimming
activity and infer that she feels a positive emotion. However, considering her
perspective on the invisible consequences of the diving activity, including
the possibility of her daughter falling off the diving board, a more plausible
emotion would be fear. Humans have the ability to deftly combine their fast
and slow thinking Mojasevic (2016) to perform emotional theory of mind. This
phenomenon is also known under the concepts of affective and cognitive empathy
Reniers et al. (2011) which relate to primary and secondary emotions. Vision
language models with contextual language abilities, such as LLAVA Liu et al.
(2023), may unlock the potential for more accurate image understanding for the
theory of mind problem, since they can combine both fast visual processing and
slower, cognitive and linguistic processing (see Fig. 1).

Indeed, our ability to recognize emotions allow us to understand one another,
build successful long-term social relationships, and interact in socially
appropriate ways. Equipping our technologies with emotion recognition
capabilities can help us improve and facilitate human-machine interactions
Picard (2000). However, emotion recognition systems today still suffer from
poor performance Barrett et al. (2019) due to the complexity of the task. This
innate and seemingly effortless capability requires understanding of the
causal relations, contextual information, social relationships as well as
using theory of mind for us to infer why someone might be feeling that way.
Many image-based emotion recognition systems focus solely on using facial or
body features Pantic and Rothkrantz (2000); Schindler et al. (2008), which can
have low accuracy in the absence of contextual information Barrett et al.
(2011); Barrett (2017).

The Affective Computing research community has been moving towards creating
datasets and building models that includes or make use of contextual
information. The EMOTIC dataset is a recent example that was introduced to
exemplify this problem, by including contextual and environmental factors to
the recognition of emotions in still images Kosti et al. (2019). It is found
that the inclusion of this contextual information beyond facial features
significantly improves the accuracy of the emotion recognition models Le et
al. (2022); Mittal et al. (2020); Wang et al. (2022). However, making use of
this information to infer the emotions of others requires commonsense
knowledge and high-level cognitive capabilities such as reasoning and theory
of mind Ong et al. (2019).

Large Language Models (LLMs) that are based on the Transformer architecture
Vaswani et al. (2017) have been recently shown to excel at Natural Language
Processing (NLP) tasks Brown et al. (2020); Chowdhery et al. (2022) and could
provide us a way to achieve emotional theory of mind through linguistic
descriptors. Providing an efficient way of processing sequenced data, and
further introducing additional methods based on transformer encoder/decoder
structures and pre-training techniques Devlin et al. (2018); Radford et al.
(2019), LLMs gained success in increasing accuracy and efficiency in NLP
problems including multimodal tasks such as Visual Question Answering Antol et
al. (2015) and Caption Generation Vinyals et al. (2015). Recently, they have
been also used in commonsense reasoning Sap et al. (2019); Bisk et al. (2020);
Li et al. (2022), emotional inference Mao et al. (2022) and theory of mind Sap
et al. (2022) tasks, however their capabilities on emotional theory of mind in
visual emotion recognition tasks have not been explored.

Until now, images of humans experiencing emotions have been analyzed based on
immediately visible characteristics (e.g. face, body, activity and physical
relationships) Kosti et al. (2019), with results that struggle to access
second order thinking of non-visible consequences. In this paper, we focus on
a multi-label, contextual emotional theory of mind task by creating a pipeline
that includes caption generation and LLMs. Towards visual grounding of text,
we use a vision language model (CLIP) to generate explainable linguistic
descriptions of a target person in images using physical signals, action
descriptors and location information. We then use an LLM (GPT-3.5) to reason
about the generated narrative text and predict the possible emotions that
person might be feeling.

The contributions of this paper are as follows:

  * •

Introduce _narrative captions_ (NarraCaps), an interpretable textual
representation for images of people experiencing an emotion;

  * •

Compared a “Fast” (CLIP) visual framework to "Fast and Slow" visual and
language reasoning frameworks (NarraCaps + LLM, ExpansionNet + LLM, LLaVA) for
contextual emotional understanding

  * •

Evaluate more than 850 social cues to construct narrative captions and perform
a large-scale visual emotional theory of mind task.

##  2 Related Work

Most of the prior work on image emotion recognition Ko (2018); Mellouk and
Handouzi (2020) focuses on human faces, using facial action coding units
(FACS) Ekman and Friesen (1976) that make use of geometry based or appearance
features to predict emotions in usually a few number of emotion categories
such as happy, sad, angry Ekman and Friesen (1971). Others also focus on body
posture Schindler et al. (2008). However, the affective computing community
has been acknowledging that facial and posture information alone would not be
sufficient to understand the emotional state of a person, and humans rely on
contextual information to understand the social and causal relationship
Barrett et al. (2011). Different from estimating the affect or sentiment
related to images including in art, landscapes, and objects Zhao et al.
(2021); You et al. (2016); Achlioptas et al. (2021); Zhao et al. (2016); Yang
et al. (2021), the task of visual emotional theory of mind is distinct and its
mechanisms poorly understood even within the affective science and social
psychology community.

_Emotional theory of mind in context._ Subsequently, work in emotional theory
of mind (in this paper, also referred to as emotion recognition) has been
focusing on the inclusion of contextual information in addition to the facial
or posture information in recent years. Early datasets such as HAPPEI Dhall et
al. (2013) proposed, for instance, emotion annotation for groups of people.
More recently, the EMOTIC dataset Kosti et al. (2019) was developed as
multilabel dataset containing 26 categories of emotions, 18,316 images and
23,788 annotated people. The related emotion recognition task is to provide a
list of emotion labels that matches those chosen by 3 annotators, responding
to the question of, “How does the person in the bounding box feel?". In
approximately 25% of the person targets, the face was not visible,
underscoring the role of context in estimating the emotion of a person in the
image. The phrase "emotional theory of mind" is used here to clarify that we
are not estimating the sentiment or emotional content of an image, but
estimating the emotion of a particular person contained in the image.

_Vision-based approaches for contextual emotion estimation_. A number of
computer vision approaches have been developed in response to the release of
the EMOTIC dataset. The EMOTIC baseline estimation method used a CNN to
extract features from the target human, as well as global features over the
entire image, and incorporate the two sources using a fusion network Kosti et
al. (2019). This resulted in a mean average precision (mAP) of 28.33 over all
26 categories. Subsequent fusion methods incorporated body and context visual
information Huang et al. (2021) at global or local scales Le et al. (2022),
investigated contextual videos Lee et al. (2019), or worked to improve subsets
of the EMOTIC dataset, such as Thuseethan et al. (2021) which considered the
photos only including two people. In PERI Mittel and Tripathi (2023), the
modulation of attention at various levels within their feature extraction
network using feature masks, resulted in an mAP of 33.33. To the extent of our
knowledge, the current best approach was Emoticon proposed by Mittal et al.
(2020), which explored the use of graph convolutional networks and image depth
maps, the found that the use of depth maps could improve the mAP to 35.48.
Overall, the latest results leave room for improvement, especially among
negative emotions including anger, aversion, pain, and embarassment.

_Large language models and theory of mind._ Recent investigations into large
language models (LLMs) have uncovered some latent capabilities of LLMs for
social intelligence, including some subtasks on emotion inference Mao et al.
(2022). Emotional theory of mind tasks using language tend to focus on
appraisal based reasoning on emotions, inferring based on a sequence of
events. For instance, among other social intelligence tasks, Sap et al. (2022)
explored how a language model could respond to an emotional event, e.g.
“Although Taylor was older and stronger, they lost to Alex in the wrestling
match. How does Alex feel?" Their findings suggested some level of social and
emotional intelligence in LLMs, finding the best performance with
GPT-3-davinci. While emotion and mental state inference remains an elusive
goal for AI systems Choi (2022), this compelling result suggests an avenue for
exploring the contextual emotion recognition problem using natural language as
an intermediary.

_Vision language models._ One natural approach to link the visual emotional
theory of mind task to the capabilities language language models is to explore
the large body of work in image captioning Stefanini et al. (2022) and visual
question and answering Zhu et al. (2016). Early work related to emotional
captions included Senticap Mathews et al. (2016) and Semstyle Mathews et al.
(2018) which focused on the emotional style of the resulting caption, rather
than an emotional theory of mind task. For example, a caption with sentiment
generated by Senticap changed the neutral phrase, “A motorcycle parked behind
a truck on a green field." to “A beat up, rusty motorcycle on unmowed grass by
a truck and trailer." More recent self-supervised vision language models such
as Visual-GPT Chen et al. (2022) for image captioning and CLIP Radford et al.
(2021); Cornia et al. (2021) provide “unreasonably effective" zero-shot
captioning Barraco et al. (2022) on datasets of images and their captions
including images of everyday activities and objects. Amidst the success of
visual language models, the opportunity still remains to generate captions
that describe details relevant to understanding the emotional state of the
person in the image.

_Natural language and emotional theory of mind._ Language is a fundamental
element in emotion and it plays a crucial role in emotion perception Lindquist
et al. (2015); Lieberman et al. (2007); Lindquist et al. (2006); Lindquist and
Gendron (2013); Gendron et al. (2012). In NLP, significant work has focused on
automatic sentiment detection that focuses on text polarity (positive or
negative); a review is available in Nandwani and Verma (2021), and is out of
the scope of this paper. Instead, we focus on the body of work in English
literature that discusses how writers use empathy to describe characters,
their actions and external cues, and “seek to evoke feelings in readers
employing the powers of narrativity." Keen and Keen (2015). Writing textbooks
such as The Emotion Thesaurus Puglisi and Ackerman (2019) provide suggestions
for sets of visual cues or actions that support emotional evocation, as a
guide for allowing readers to imagine the most relevant features of the person
in the scene. For example, to evoke curiosity (see Table 7), a writer may
narrate, "she tilted the head to the side, leaning forward, eyebrows
furrowing" or "she raised her eyebrows and her body posture perked up." To the
best of our knowledge, a gap in the emotional theory of mind task remains in
testing how large language models (LLMs) and language-vision models (LVMs)
could be used to describe and reason upon a visual snapshot of a person
experiencing an emotion at a given moment in time, rather than reasoning about
a sequence of events.

_Thinking fast and slow._ Cognitive processes can be categorized into two main
groups, commonly referred to as intuition and reason, or System 1 and System 2
Mojasevic (2016). In this dual-process model, one system rapidly generates
intuitive answers to judgment problems as they arise, while the other system
monitors the quality of these proposals using reasoning Lieberman (2007);
Evans (2008); Kahneman et al. (2002). Regarding empathy, a general term
encompassing various neurocognitive functions, the data suggest that a two-
factor structure (affective and cognitive empathy) provides the best and most
parsimonious fit Reniers et al. (2011); Blair (2005).

##  3 Methodology

Table 1: Qualitative results of EMOTIC images, ground truth (GT) labels, captions and inferred labels. |  |
---|---|---
|  1\. GT: Anticipation, Excitement, Happiness, Pleasure, Sympathy
---
NarraCap: This person is an elderly person who is petting an animal (not a
cat) at a nursing home. This person is offering encouragement and support.
GPT-3.5: Confidence, Affection, Engagement, Happiness, Peace, Esteem.
CLIP: Doubt/Confusion, Esteem, Sensitivity, Engagement, Sympathy, Affection
LLaVA: Happiness, Affection, Surprise
|  2\. GT: Engagement, Happiness, Peace
---
NarraCap: This person is an adult who is hugging at a airport. This person is
clutching at another person for support. This person is pulling someone into a
side hug. This person is putting an arm around someone’s shoulders.
GPT-3.5: Confidence, Engagement, Happiness, Peace, Pleasure, Esteem
CLIP: Sensitivity, Disconnection, Happiness, Suffering, Surprise, Affection
LLaVA: Happiness, Excitement, Sadness
|  3\. GT: fatigue, peace
---
NarraCap: This person is an adult who is sleeping at a library. This person is
curling up to take up less space. This person is falling asleep in odd places
at odd times.
GPT-3.5: Fear, Fatigue, Embarrassment, Disconnection, Sadness, Sympathy
CLIP: Sympathy, Engagement, Suffering, Disquietment, Disconnection, Fatigue
LLaVA: -
|  |
|  4\. GT: Confidence, Esteem, Excitement, Happiness, Pleasure
---
NarraCap: This person is an adult who is skiing at a ski resort. This person
has hands raised into the air. This person is waving the arms, using grand
gestures.
GPT-3.5: Confidence, Engagement, Happiness, Pleasure, Excitement, Anticipation
CLIP: Esteem, Doubt/Confusion, Surprise, Happiness, Confidence, Fear
LLaVA: Happiness, Excitement, Sadness
|  5\. GT: Happiness, Peace, Pleasure
---
NarraCap: This person is an adult who is surfing water at a beach. This person
is participating in relaxing activities.
GPT-3.5: Confidence, Engagement, Happiness, Peace, Pleasure, Esteem
CLIP: Happiness, Sympathy, Embarrassment, Peace, Confidence, Esteem
LLaVA: Happiness, Confidence, Excitement, Pain
|  6\. GT: Anticipation, Confidence, Doubt/Confusion, Engagement, Excitement,
Happiness
---
NarraCap: This person is an adult who is snowboarding at a ski resort. This
person has shoulders lowered and loose. This person is pulling arms and legs
in toward the core.
GPT-3.5: Confidence, Engagement, Happiness, Pleasure, Excitement, Anticipation
CLIP: Pain, Surprise, Doubt/Confusion, Confidence, Fear, Suffering
LLaVA: Fear, Sadness
|  |
|  7\. GT: Engagement, Sadness, Suffering, Sympathy
---
NarraCap: This person is an adult who is crying at a refugee camp. This person
is crying, wailing, begging for help.
GPT-3.5: Fear, Sadness, Pain, Anger, Disquietment, Sympathy
CLIP: Anger, Fatigue, Affection, Sensitivity, Sadness, Sympathy
LLaVA: Sadness, Sympathy, Fear
|  8\. GT: Happiness, Pleasure
---
NarraCap: This person is an adult who is shaking hands at a board room. This
person is engaging excitedly with others. This person is using a strong,
businesslike handshake.
GPT-3.5: Confidence, Engagement, Happiness, Pleasure, Excitement, Anticipation
CLIP: Suffering, Disconnection, Sensitivity, Esteem, Confidence, Engagement
LLaVA: Excitement, Anticipation
|  9\. GT: Affection, Engagement, Happiness, Pleasure
---
NarraCap: This person is a baby who is grooming a dog at a vet clinic. This
person is displaying affection with friends or loved ones. This person is
gazing at someone, full of love.
GPT-3.5: Affection, Happiness, Peace, Pleasure, Esteem, Excitement
CLIP: Sadness, Engagement, Peace, Happiness, Pleasure, Affection
LLaVA: Happiness, Affection, Surprise, Sympathy
Table 2: Failure examples of narrative captions. |  |
---|---|---
|  1: This person is an adult who is acting in a play at a therapist office.
This person is not engaging in conversation.
---
|  2: This person is a baby who is punching a person (boxing) at a sporting
event stands. This person is chewing on a knuckle. This person is propping the
head up with fist. This person is responding with anger by punching.
---
|  3: This person is an elderly person who is doing a tennis swing at a rec
center. This person has leg muscles tightening. This person is using the body
to nudge, push, or block.
---
|  |
|  4: This person is an adult who is longboarding at a skate park. This person
has fingers loosely clasped in one’s lap. This person has raised, prominent
cheekbones from smiling. This person is rubbing one’s hands on pant legs.
---
|  5: This person is an elderly person who is huddling at a city bus. This
person is crying, wailing, begging for help. This person is squeezing the eyes
shut, refusing to look.
---
|  6: This person is an adult who is having a pillow fight at a run-down
apartment. This person is tossing and turning in bed, an inability to sleep.
---

We aim to compare zero-shot methods across the varying model families offered
by OpenAI111https://openai.com/, including a zero-shot classifier (CLIP
Radford et al. (2021)), a large language model (GPT-3), and a large vision
language model. As GPT-4 was inaccessible to the authors due to waitlist
access, we select LLaVA for its vision language ability, which according to
Liu et al. (2023) was trained using GPT-4 and is state-of-the-art alternative
for the vision-language instruct task. In this sense, we focus on comparing
the breadth of approaches in large language model types that could perform
this task, rather than comparing between competing open-source or closed
source models within each family (e.g. GPT-3 vs. LLAMA).

In order to access the natural language capabilities of GPT-3.5, we first aim
to generate a caption of the image such that the text representation allows
for understanding the emotion of the person in it, then GPT-3.5 for language
reasoning. We use the vision language model CLIP Radford et al. (2021), which
can be used for out-of-the-box image classification for this task, along with
a method we call Narrative Captioning.

###  3.1 Narrative Captioning

We build upon previous captioning work that describes a person and their
activity. First, given an image with the bounding box of a person, we extract
the cropped bounding box and pass it along with the gender/age category (girl,
boy, man, woman, older man, older woman) to CLIP to understand who is in the
picture. Next, we pass the entire image to understand the what is happening in
the image by analyzing the actions. The action list is extracted from the
Kinetics-700 Smaira et al. (2020), UCF-101 Soomro et al. (2012), and HMDB
datasets Kuehne et al. (2011).

The contribution of the present narrative caption is to add the how aspect of
the image. We obtained over 1000 social signals from a guide to writing about
emotion Puglisi and Ackerman (2019). We then filtered these signals to include
only those visible in an image, resulting in more than 850 social signals. By
passing these signals and the cropped bounding box to CLIP, we generate
captions that describe the person in the image. To provide additional context,
we use 224 environmental descriptors from a writer’s guide to urban Puglisi
and Ackerman (2016b) and rural Puglisi and Ackerman (2016a) settings to
describe where the person in the scene is located.

The prompts we selected for CLIP are as follows: ’The person in the image
is/has [gender/action/physical signals]’ for gender, action, and physical
signals, and ’The image is happening in a [location]’ for locations. Examples
of narrative captions (NarraCaps) can be found in Table 1, as well as failure
cases in Table 2.

Table 3:  Results for different approaches of picking physical signals on the
$1000$ randomly picked people from validation set. We also report the average
number of physical signals and average number of words in captions for each
approach.

| mAP | ave #sigs | ave cap len |
---|---|---|---|---
top1 | 25.22 | 1.0 | 22.016 |
top3 | 25.33 | 3.0 | 37.24 |
top5 | 25.35 | 5.0 | 52.506 |
$mean+std$ | 25.43 | 44.512 | 330.15 |
$mean+3\times{std}$ | 25.53 | 12.127 | 104.702 |
$mean+5\times{std}$ | 25.32 | 5.524 | 56.184 |
$mean+7\times{std}$ | 25.47 | 3.058 | 37.816 |
$mean+9\times{std}$ | 25.54 | 1.899 | 28.986 |

How we should pick the physical signals to describe how? To answer this
question we evaluated different thresholding methods on the validation set.
After obtaining probabilities from CLIP, we picked the top 1, 3, or 5 physical
signals with highest probabilities to generate captions and record the mAP. As
the second approach, we picked all the signals which had the probability
greater than $Mean+std$, $Mean+2\times{std}$, $Mean+5\times{std}$,
$Mean+7\times{std}$, and $Mean+9\times{std}$ as valid physical signals and
used them to generate the captions. Note that the mean in this case is equal
to $100$ divided by number of physical signals. As you can see in Table 3, the
best results are for the when pick the top $Mean+9\times{std}$ physical
signals.

###  3.2 Text to Emotion Inference

Using the caption, we then obtain a set of emotions experienced by this
individual in the bounding box using a large language model. To do this, we
provide the caption, along with a prompt, to GPT-3.5 (text-davinci-003) with
the temperature set to zero. The prompt describes the emotions precisely as
described in Kosti et al. (2019) and asks for the best six emotion labels
understood from the narrative caption, denoted by <caption>:

"<caption> From suffering (which means psychological or emotional suffering;
distressed; anguished), pain (which means physical pain), aversion (which
means feeling disgust, dislike, repulsion; feeling hate), disapproval (which
means feeling that something is wrong or reprehensible; contempt; hostile),
[…] sadness (which means feeling unhappy, sorrow, disappointed, or
discouraged), and sympathy (which means state of sharing others’ emotions,
goals or troubles; supportive; compassionate), pick a set of six most likely
labels that this person is feeling at the same time."

Six labels were requested from GPT-3.5 since the average number of ground
truth labels in the validation set was 6.

For generalization and reproducibility purposes, we also tested our narrative
captions with three open-source Large Language Models: Baichuan-
chat-13B222https://github.com/baichuan-inc/Baichuan-13B, Falcon-Instruct-7B
Almazrouei et al. (2023), and MPT-Chat-7B Team et al. (2023). However, due to
their significantly lower number of parameters compared to GPT-3.5, they did
not perform as well. These models achieved mAP scores of $17.12$, $17.10$, and
$17.11$, respectively, which are considerably lower than GPT-3.5’s score of
$18.58$.

##  4 Experiments

Table 4: Performance of different models on EMOTIC. Action, NarraCap, and
ExpNet captioning use GPT-3 for inference.

Emotions | Rand | Maj | Rand(W) | CLIP | Action | LLaVA | ExpNet | NarraCap
---|---|---|---|---|---|---|---|---
Affection | 13.27 | 13.41 | 13.58 | 22.63 | 18.13 | 22.97 | 18.51 | 19.61
Anger | 2.45 | 2.45 | 2.42 | 6.09 | 2.85 | 6.04 | 2.87 | 5.60
Annoyance | 4.74 | 4.89 | 4.91 | 5.14 | 5.75 | 5.27 | 6.42 | 6.32
Anticipation | 46.86 | 47.05 | 47.23 | 47.31 | 48.91 | 48.84 | 50.18 | 48.59
Aversion | 3.05 | 3.05 | 3.07 | 3.03 | 3.38 | 3.05 | 3.83 | 3.28
Confidence | 47.33 | 47.43 | 47.84 | 52.12 | 51.78 | 48.43 | 55.16 | 49.66
Disapproval | 4.61 | 4.55 | 4.54 | 4.44 | 4.77 | 4.63 | 4.93 | 4.78
Disconnection | 15.80 | 15.54 | 15.34 | 15.42 | 15.82 | 15.52 | 17.09 | 17.48
Disquietment | 12.45 | 12.43 | 12.25 | 13.40 | 12.70 | 12.45 | 12.86 | 13.27
Doubt/Confusion | 14.31 | 14.52 | 14.60 | 14.58 | 14.60 | 14.52 | 14.51 | 15.21
Embarrassment | 1.89 | 1.88 | 1.85 | 1.96 | 2.07 | 2.02 | 2.31 | 2.03
Engagement | 77.28 | 77.33 | 77.24 | 76.57 | 79.02 | 78.04 | 79.38 | 78.36
Esteem | 13.33 | 13.64 | 14.03 | 13.75 | 13.81 | 13.64 | 13.67 | 13.87
Excitement | 47.80 | 47.94 | 47.34 | 48.31 | 53.31 | 54.29 | 56.62 | 55.06
Fatigue | 6.02 | 5.91 | 5.95 | 7.38 | 6.36 | 6.91 | 7.24 | 6.44
Fear | 3.27 | 3.20 | 3.20 | 3.67 | 3.56 | 4.35 | 4.03 | 3.90
Happiness | 48.05 | 47.95 | 48.27 | 52.60 | 48.93 | 53.41 | 49.40 | 52.51
Pain | 2.03 | 1.96 | 1.95 | 1.99 | 2.11 | 2.32 | 2.15 | 2.28
Peace | 14.70 | 14.61 | 14.33 | 16.38 | 16.15 | 15.25 | 16.72 | 16.42
Pleasure | 29.17 | 28.99 | 29.35 | 30.74 | 29.50 | 29.07 | 30.30 | 31.72
Sadness | 5.32 | 5.34 | 5.40 | 8.20 | 6.22 | 8.62 | 6.97 | 8.23
Sensitivity | 2.88 | 2.93 | 2.94 | 3.42 | 2.95 | 2.91 | 2.97 | 2.93
Suffering | 3.50 | 3.50 | 3.47 | 3.06 | 3.50 | 3.58 | 3.50 | 3.50
Surprise | 6.34 | 6.14 | 6.15 | 6.87 | 6.11 | 6.80 | 6.13 | 6.11
Sympathy | 8.33 | 8.45 | 8.34 | 9.01 | 8.52 | 8.92 | 9.42 | 9.35
Yearning | 6.48 | 6.47 | 6.45 | 6.40 | 6.58 | 6.46 | 6.50 | 6.58
mAP | 16.97 | 16.98 | 17.00 | 18.25 | 17.98 | 18.40 | 18.60 | 18.58

We use the EMOTIC dataset Kosti et al. (2019) which covers 26 different labels
as shown in Table 4. The related emotion recognition task is to provide a list
of emotion labels that matches those chosen by annotators. The training set
(70%) was annotated by 1 annotator, where Validation (10%) and Test (20%) sets
were annotated by 5 and 3 annotators, respectively.

###  4.1 Evaluation Metrics and Baselines

Following the previous work in context-based emotion recognition, we use the
standard Mean Average Precision (mAP) metric. We compare the following
methods.

EMOTIC Along with the EMOTIC dataset, Kosti et al. (2019) introduced a two-
branched network baseline. The first branch is in charge of extracting body
related features and the second branch is in charge of extracting scene-
context features. Then a fusion network combines these features and estimates
the output.

Emoticon Motivated by Frege’s principle Resnik (1967), Mittal et al. (2020)
proposed an approach by combining three different interpretations of context.
They used pose and face features (context1), background information
(context2), and interactions/social dynamics (context3). They used a depth map
to model the social interactions in the images. Later, they concatenate these
different features and pass it to fusion model to generate outputs. Mittal et
al. (2020)

LLaVA Large Language and Vision Assistant (LLaVA) Liu et al. (2023) is a
multi-purpose multimodal model designed by combining CLIP’s visual encoder
Radford et al. (2021) and LLAMA’s language decoder Touvron et al. (2023). The
model is fine-tuned end-to-end on the language-image instruction-following
data generated using GPT-4 OpenAI (2023).

Random We consider selecting either $6$ emotions randomly from all possible
labels (Rand) or selecting $6$ labels randomly where the weights are
determined by the number of times each emotion is repeated in the validation
set (Rand(W)).

Majority This Majority baseline selects the top $6$ most common emotions in
the validation set as the predicted labels for all test images (Maj).

CLIP-only We evaluate the capabilities of the vision language model CLIP to
predict the emotion labels. In this study we pass the image with the emotion
labels in the format of: "The person in the red bounding box is feeling
{emotion label}" to produce the probabilities that CLIP gives to each of these
sentences. We then pick the $6$ labels with the highest probabilities.

Action-only We consider only the what portion of the caption and disregard
physical signals, environment and gender. We pass the set of actions to CLIP
and generate a single action caption per image in the form of This person is
[activity]. We then send this caption to GPT-3.5 and obtain 6 possible
emotions using the same prompt template as described in Sec. 2B.

ExpansionNet v2 Hu et al. (2022) is a fast end-to-end training model for Image
Captioning. Its characteristic is expanding the input sequence into another
length with a different length and retrieving the original length back again.
The model achieves the state of art performance over the MS-COCO 2014
captioning challenge, and was used as a backbone for a recent approach trained
on EMOTIC de Lima Costa et al. (2023). We fed the captions generated by
ExpansionNet v2 into GPT-3.5 to obtain six emotion labels (ExpNet).

Narrative captioning and LLM Using narrative captions generated using CLIP, we
pass them to Large Language Models and ask for the related emotion labels
using a proper prompt. For LLMs, we used GPT-3.5, Baichuan-
chat-13B333https://github.com/baichuan-inc/Baichuan-13B, Falcon-Instruct-7B
Almazrouei et al. (2023), and MPT-Chat-7B Team et al. (2023). GPT-3.5 has the
best performance compared to the rest, which is predictable given its 175B
parameters.

###  4.2 Results and Analyses

The results for zero shot methods are shown in Table 4, and example images
with captions in Table 1.

Figure 2: Results including standard error depict significantly higher mAP for
"Fast and Slow" methods compared to "Fast" only
(CLIP).$18$$18.2$$18.4$$18.6$$18.8$mAPCLIPLLaVAExpNet + GPT3.5NarraCaps +
GPT3.5

How do different strategies affect the results?

Among the zero-shot methods, as shown in Fig. 2, we can observe that the
methods combining "Fast and Slow" frameworks (NarraCaps+GPT3.5, ExpNet+GPT3.5,
and LLaVA) outperform the methods relying solely on the "Fast" framework
(CLIP). For instance, in Table 1.4, it appears that CLIP perceives the body
pose with raised arms and predicts the emotions of ‘surprise’ and ‘fear’. On
the other hand, the ’Fast and Slow’ frameworks, given the context of skiing,
may reason that this body pose is not related to fear or surprise but is more
indicative of ‘happiness’ and ‘excitement’. Similarly, in Table 1.5, CLIP
predicts embarrassment for the woman at the beach. After investigation, it
appears that CLIP tends to associates images containing bare skin with
embarrassment. In reasoning about the location of the beach, however, the
language models no longer predict embarassment.

We note that EMOTIC and Emoticon, which utilized the EMOTIC training dataset
to fine-tune their models, achieved better performance with mAP scores of
$27.38$ and $35.48$, respectively, in comparison to others that did not
directly use the EMOTIC training set, employing zero-shot methods.

How do different prompts affect the results? It is important to carefully
select an appropriate prompt for GPT-3.5. The initial prompt for GPT-3.5 was
the generated caption plus "From suffering, pain, aversion, disapproval,
anger, fear, annoyance, fatigue, disquietment, doubt/confusion, embarrassment,
disconnection, affection, confidence, engagement, happiness, peace, pleasure,
esteem, excitement, anticipation, yearning, sensitivity, surprise, sadness,
and sympathy, pick top labels that describe the emotion of this person." We
randomly picked $1000$ samples of validation data, and for each sample
provided GPT-3 its generated narrative caption plus prompt. We obtained a mAP
equal to $25.18$. After changing the prompt to "pick a set of six most likely
labels that this person is feeling at the same time", the mAP increased to
$25.50$. We then tried to change the GPT-3.5 prompt to add the definitions of
the labels provided by EMOTIC (see Section 3.2) and we obtained an mAP equal
to $25.54$.

How do age, gender, activity, environment, and physical signals affect the
results? One of the advantages of the NarraCaps approach is that it provides a
way to explicitly select image details to include for inference and perform
ablations using the text representation. To assess the effect of gender,
instead of using specific gender labels such as "a baby boy," "a baby girl,"
"a girl," "a boy," "a woman," "a man," "an elderly man," and "an elderly
woman," we only utilized the labels "a female" and "a male." Furthermore, to
examine gender, we modified the label list to "a baby," "a kid," "an adult,"
and "an elderly person." To investigate the impact of activity, environment,
and physical signals, we excluded those components from the captions. The
results of each study on a set of 1000 images are presented in Table 5. The
table indicates that action had the most significant influence, followed by
physical signals and environment. Including gender in the captions resulted in
a slight reduction in mAP, although based on calculations of standard error,
the difference may be attributed to noise.

Table 5: Ablation study using NarraCaps+GPT-3 on 1000 images randomly selected
from the validation set.

Ablation Settings | mAP | diff. | SE
---|---|---|---
age | gender | environment | action | physical signals |  |  |
✓ | ✓ | ✓ | ✓ | ✓ | 25.54 | - | 0.27
- | ✓ | ✓ | ✓ | ✓ | 25.46 | -0.08 | 0.25
✓ | - | ✓ | ✓ | ✓ | 25.66 | +0.12 | 0.29
✓ | ✓ | - | ✓ | ✓ | 25.22 | -0.32 | 0.26
✓ | ✓ | ✓ | - | ✓ | 24.91 | -0.63 | 0.24
✓ | ✓ | ✓ | ✓ | - | 25.1 | -0.44 | 0.24

Table 6: Test set mAP and standard error (SE) when separated into 1, 2, and
multiple people in the image.

|  | CLIP | LLaVA | ExpNet | NarraCaps
---|---|---|---|---|---
mAP | 1 | 18.61 | 18.66 | 19.50 | 19.18
2 | 18.28 | 18.61 | 18.40 | 18.66
>2 | 17.78 | 17.85 | 17.76 | 17.88
SE | 1 | 0.14 | 0.16 | 0.17 | 0.18
2 | 0.15 | 0.16 | 0.15 | 0.17
>2 | 0.14 | 0.14 | 0.14 | 0.15

How does the number of people in the image impact the mAP? We evaluate
different methods based on the number of people in the image: one, two, or
multiple people. As shown in Table 6, the mAP decreases as the number of
people increases. This reduction in performance can be attributed to the more
complex situations that arise when there are more people in the scene. Also,
human interactions are not covered in NarraCaps, and both LLaVA and CLIP face
difficulties in detecting the correct person referred to in the question. For
ExpansionNet + GPT-3.5 (ExpNet), we find that while it outperformed the
NarraCaps method for single-person images, it did not perform as well in
images with 2 or more people. This is expected, as this captioning method is
not trained to focus on a particular person.

How much improvement can we achieve by enhancing the captions? We recruited an
annotator who speaks native North American English. Using a web interface Yang
et al. (2023) (details in Appendix), we manually generated captions for 387
images that contained the 14 negative emotion categories: suffering,
annoyance, pain, sadness, anger, aversion, disquietment, doubt/confusion,
embarrassment, fear, disapproval, fatigue, disconnection, sensitivity.
Negative emotions were the focus of this experiment because they were poorly
recognized compared to positive emotions. The human-generated captions were
then input into GPT-3.5, resulting in an mAP of $20.37$. Conversely, when
utilizing our narrative captioning method in conjunction with GPT-3.5, we
obtained an mAP of $18.28$. This suggests that there is a disparity between
human-generated captions and our narrative captions, and inspecting the
differences to improve the automatic narrative caption method is an
interesting area for future work. Using the provided EMOTIC
interface444https://github.com/Tandon-A/emotic/, we evaluated EMOTIC on these
images and achieved an mAP of $19.03$. The result of this experiment suggests
promise in the GPT-3.5 reasoning task if the captions can be well defined.

Are ExpansionNet captions sufficient? ExpansionNet v2 performs comparably and
slightly better than NarraCap with GPT-3. One possible explanation lies in the
distribution of the data sources in EMOTIC, of which a large majority is from
Microsoft COCO (4397 images) Lin et al. (2014). This data source contains
scenes in the wild, focusing on objects and activities such as eating, sports,
and so on (Table 1.4-6). The EMOTIC dataset also contains collected images
retrieved from Google searches with emotion keywords, resulting in an EmoDB
subset containing 810 images with many acted emotional portrayals without a
clear location or associated activity (Table 1.7-9). On this EmoDB image
subset, ExpNet results in an mAP of 19.97 while NarraCap results in an mAP of
22.97. This suggests that the success of captioning with ExpNet may be due to
the heavy prevalence of emotion labels as a result of activities or scene
contexts in COCO, e.g. playing sports being exciting.

How important are small differences in mAP? We used a bootstrap method to
calculate standard error in mAP (Figure 2). We observe that differences in mAP
between certain methods are indeed significant in our large testset, for
example between NarraCaps/LLAVA vs. CLIP only. This makes sense as CLIP, as a
contrastive vision-language model, does not provide the long-context, slow
reasoning provided by GPT-3.5 and LLAVA.

##  5 Discussion

Our findings showed that images with a single person resulted in better
accuracy than images with two or more people in them (see Table 6). One reason
for this could be attributed to the lack of social interaction signals in our
captions. Images with more than two people had the lowest accuracy results.
The literature on group emotions suggests, the shared condition in the group
such as the environment and group information could affect the individual
emotions within the group Veltmeijer et al. (2021). In our proposed model and
linguistic descriptors, contextual information mostly relied on environmental
information, with some inclusion of group events (e.g., "black tie event",
"party"). For example, Table 2.4 shows a possible group of friends, which was
not captured by the captioning method. Further, in some cases where the
physical signals of the person in the bounding box are not readily available,
humans tend to rely on the information from the physical signals from other
individuals within the same group to infer and reason about the emotions of a
single individual of that group, which is referred as a bottom-up approach
Barsade and Gibson (1998). In our current proposal, we only use the person in
the bounding box’s for person descriptors, physical and action signals
regardless of its available information content. As an example, the figure in
Table 2.2 involves multiple individuals in the background that have occluded
images with few detectable physical signals, where reasoning about the
person’s emotions requires to seek meaningful signals from other individuals
in the group and the group’s behavior as a whole. Our proposed model currently
does not allow for these types of emotional reasoning strategies that seems to
be an important part of group emotion recognition. Moreover, for the activity
and environment detection using CLIP, we performed a standard evaluation with
a limited set of classes without a "don’t know" or null class, resulting in
some mis-captioning. For instance, as shown in Table 2.6, the model
misunderstood the action as "pillow fight", whereas the action should be
described as "covering ears with a pillow", or "yelling".

##  6 Conclusion

In this paper, we begin to explore the capabilities of vision language models
and LLMs (here, CLIP and GPT-3.5) for the visual emotional theory of mind
task. We found that zero-shot slow and fast methods (vision and language)
outperformed the fast only (vision) frameworks. However they still do not
perform better than the models trained specifically for this task on the
EMOTIC dataset. Further research could explore improving the narrative caption
by adding other contextual factors to the caption, such as human-object
interactions and relationships with other people in the image. Further studies
on the characteristics of emotionally comprehensive captioning, coupled with
GPT-3.5 or with human evaluators could be done. GPT-4 may also perform better
than LLaVA in jointly performing the reasoning and vision processing tasks.

##  7 Limitations

The present study assessing emotional theory of mind on the EMOTIC image
dataset using the notion of fast and slow processing, albeit being first of
its kind, is not without limitations. First, we only focused on assessing
zero-shot methods across the varying model families offered by OpenAI,
including a zero-shot classifier (CLIP), a large language model (GPT-3), and a
large vision language model (LLAVA). It is possible that other models in the
family of VLMs and LLM could outperform the investigated models.

In addition, this study focused on studying the as-is capabilities of these
models, without any training, fine-tuning or few-shot prompting given the
EMOTIC training dataset. There is one exception: we informed each model to
provide us 6 labels based on the average number of labels in the validation
set, but did so consistently across all experiments. Finally, we noticed that
the EMOTIC dataset, while being one of the most challenging image emotion
datasets including context, also has small imperfections, including some
bounding boxes that contain 2 people instead of only one. Although the case
described is rare, a future study could evaluate on other datasets, e.g. one
person emotion expression datasets without context, which is considered to be
a simpler task.

###  7.1 Limitations of NarraCaps + GPT-3 method

In attempting to access GPT-3’s reasoning capability for this task, we created
a new method to generate narrative captions for each person-image pair. One
limitation is that we did not perform any quantitative human evalation on the
quality of the resulting captions themselves, rather focusing on end-to-end
performance against the original human annotated baseline. NarraCaps also did
not describe the social interactions or interactions with objects, which if
added may increase performance.

Another limitation of this method is the English language and culture based
thesaurus of social signals, written by an North American author. Non-verbal
social signals are culture-dependent Tanaka et al. (2010) and translation may
not suffice. An interesting direction for future work could investigate the
distribution of social signals across emotions and between cultures.

##  8 Ethical Considerations

The research team conducted all the coding and emotion annotation for the
images in this study. The images were sourced from the EMOTIC Dataset, which
consists images obtained from internet and public datasets like MSCOCO and
Ade20k. To access the EMOTIC dataset, permission must be obtained from the
owners. Research team has no affiliation with the individuals depicted in the
images. The images show individuals experiencing a range of emotions, such as
grief at a funeral, participation in protests, or being in war-related
scenarios. However, the dataset does not contain images that surpass the
explicitness or intensity typically encountered in news media.

It is crucial to note that there might be biases affecting this work, such as
cultural biases from annotators and biases caused by the languages that
GPT-3.5 and CLIP are trained on. Furthermore, The source Emotion Thesaurus
book is written by North American authors, and similar writing guides in other
languages may lead to differing results.

The implementation of this work utilized pre-trained models, like GPT-3.5,
CLIP, and LLaVA. Although no additional training was conducted in this study,
it is crucial to acknowledge the significant carbon cost associated with
training such models, which should not be underestimated.

As a final note, it is worth mentioning that the authors strongly oppose the
utilization of any software that employs emotion estimation approaches with
the potential to violate privacy, personal autonomy, and human rights.

## References

  * Achlioptas et al. (2021) Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas. 2021.  Artemis: Affective language for visual art.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11569–11579.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, et al. 2023.  Falcon-40b: an open large language model with state-of-the-art performance.  Technical report, Technical report, Technology Innovation Institute.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433.
  * Barraco et al. (2022) Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022.  The unreasonable effectiveness of clip features for image captioning: an experimental analysis.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4662–4670.
  * Barrett (2017) Lisa Feldman Barrett. 2017.  _How emotions are made: The secret life of the brain_.  Pan Macmillan.
  * Barrett et al. (2019) Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M Martinez, and Seth D Pollak. 2019.  Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements.  _Psychological science in the public interest_ , 20(1):1–68.
  * Barrett et al. (2011) Lisa Feldman Barrett, Batja Mesquita, and Maria Gendron. 2011.  Context in emotion perception.  _Current directions in psychological science_ , 20(5):286–290.
  * Barsade and Gibson (1998) Sigal G Barsade and Donald E Gibson. 1998.  Group emotion: A view from top and bottom.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 7432–7439.
  * Blair (2005) Robert James R Blair. 2005.  Responding to the emotions of others: Dissociating forms of empathy through the study of typical and psychiatric populations.  _Consciousness and cognition_ , 14(4):698–718.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2022) Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022.  Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18030–18040.
  * Choi (2022) Yejin Choi. 2022.  The curious case of commonsense intelligence.  _Daedalus_ , 151(2):139–155.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Cornia et al. (2021) Marcella Cornia, Lorenzo Baraldi, Giuseppe Fiameni, and Rita Cucchiara. 2021.  Universal captioner: Inducing content-style separation in vision-and-language model training.  _arXiv preprint arXiv:2111.12727_.
  * de Lima Costa et al. (2023) Willams de Lima Costa, Estefania Talavera, Lucas Silva Figueiredo, and Veronica Teichrieb. 2023.  High-level context representation for emotion recognition in images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 326–334.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dhall et al. (2013) Abhinav Dhall, Jyoti Joshi, Ibrahim Radwan, and Roland Goecke. 2013.  Finding happiest moments in a social context.  In _Computer Vision–ACCV 2012: 11th Asian Conference on Computer Vision, Daejeon, Korea, November 5-9, 2012, Revised Selected Papers, Part II 11_ , pages 613–626. Springer.
  * Ekman and Friesen (1971) Paul Ekman and Wallace V Friesen. 1971.  Constants across cultures in the face and emotion.  _Journal of personality and social psychology_ , 17(2):124.
  * Ekman and Friesen (1976) Paul Ekman and Wallace V Friesen. 1976.  Measuring facial movement.  _Environmental psychology and nonverbal behavior_ , 1:56–75.
  * Evans (2008) Jonathan St BT Evans. 2008.  Dual-processing accounts of reasoning, judgment, and social cognition.  _Annu. Rev. Psychol._ , 59:255–278.
  * Gendron et al. (2012) Maria Gendron, Kristen A Lindquist, Lawrence Barsalou, and Lisa Feldman Barrett. 2012.  Emotion words shape emotion percepts.  _Emotion_ , 12(2):314.
  * Hu et al. (2022) Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. 2022.  Expansionnet v2: Block static expansion in fast end to end training for image captioning.  _arXiv preprint arXiv:2208.06551_.
  * Huang et al. (2021) Yibo Huang, Hongqian Wen, Linbo Qing, Rulong Jin, and Leiming Xiao. 2021.  Emotion recognition based on body and context fusion in the wild.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3609–3617.
  * Kahneman et al. (2002) Daniel Kahneman, Shane Frederick, et al. 2002.  Representativeness revisited: Attribute substitution in intuitive judgment.  _Heuristics and biases: The psychology of intuitive judgment_ , 49(49-81):74.
  * Keen and Keen (2015) Suzanne Keen and Suzanne Keen. 2015.  Narrative emotions.  _Narrative Form: Revised and Expanded Second Edition_ , pages 152–161.
  * Ko (2018) Byoung Chul Ko. 2018.  A brief review of facial emotion recognition based on visual information.  _sensors_ , 18(2):401.
  * Kosti et al. (2019) Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. 2019.  Context based emotion recognition using emotic dataset.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(11):2755–2766.
  * Kuehne et al. (2011) Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. 2011.  Hmdb: a large video database for human motion recognition.  In _2011 International conference on computer vision_ , pages 2556–2563. IEEE.
  * Le et al. (2022) Nhat Le, Khanh Nguyen, Anh Nguyen, and Bac Le. 2022.  Global-local attention for emotion recognition.  _Neural Computing and Applications_ , 34(24):21625–21639.
  * Lee et al. (2019) Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. 2019.  Context-aware emotion recognition networks.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10143–10152.
  * Li et al. (2022) Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d’Autume, Phil Blunsom, and Aida Nematzadeh. 2022.  A systematic investigation of commonsense knowledge in large language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11838–11855.
  * Lieberman (2007) Matthew D Lieberman. 2007.  Social cognitive neuroscience: a review of core processes.  _Annu. Rev. Psychol._ , 58:259–289.
  * Lieberman et al. (2007) Matthew D Lieberman, Naomi I Eisenberger, Molly J Crockett, Sabrina M Tom, Jennifer H Pfeifer, and Baldwin M Way. 2007.  Putting feelings into words.  _Psychological science_ , 18(5):421–428.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer.
  * Lindquist et al. (2006) Kristen A Lindquist, Lisa Feldman Barrett, Eliza Bliss-Moreau, and James A Russell. 2006.  Language and the perception of emotion.  _Emotion_ , 6(1):125.
  * Lindquist and Gendron (2013) Kristen A Lindquist and Maria Gendron. 2013.  What’s in a word? language constructs emotion perception.  _Emotion Review_ , 5(1):66–71.
  * Lindquist et al. (2015) Kristen A Lindquist, Jennifer K MacCormack, and Holly Shablack. 2015.  The role of language in emotion: Predictions from psychological constructionism.  _Frontiers in psychology_ , 6:444.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Mao et al. (2022) Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2022.  The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection.  _IEEE Transactions on Affective Computing_.
  * Mathews et al. (2016) Alexander Mathews, Lexing Xie, and Xuming He. 2016.  Senticap: Generating image descriptions with sentiments.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30.
  * Mathews et al. (2018) Alexander Mathews, Lexing Xie, and Xuming He. 2018.  Semstyle: Learning to generate stylised image captions using unaligned text.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Mellouk and Handouzi (2020) Wafa Mellouk and Wahida Handouzi. 2020.  Facial emotion recognition using deep learning: review and insights.  _Procedia Computer Science_ , 175:689–694.
  * Mittal et al. (2020) Trisha Mittal, Pooja Guhan, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020.  Emoticon: Context-aware multimodal emotion recognition using frege’s principle.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14234–14243.
  * Mittel and Tripathi (2023) Akshita Mittel and Shashank Tripathi. 2023.  Peri: Part aware emotion recognition in the wild.  In _Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI_ , pages 76–92. Springer.
  * Mojasevic (2016) Aleksandar S Mojasevic. 2016.  Daniel kahneman: Thinking, fast and slow.
  * Nandwani and Verma (2021) Pansy Nandwani and Rupali Verma. 2021.  A review on sentiment analysis and emotion detection from text.  _Social Network Analysis and Mining_ , 11(1):81.
  * Ong et al. (2019) Desmond C Ong, Jamil Zaki, and Noah D Goodman. 2019.  Computational models of emotion inference in theory of mind: A review and roadmap.  _Topics in cognitive science_ , 11(2):338–357.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Pantic and Rothkrantz (2000) Maja Pantic and Leon JM Rothkrantz. 2000.  Expert system for automatic analysis of facial expressions.  _Image and Vision Computing_ , 18(11):881–905.
  * Picard (2000) Rosalind W Picard. 2000.  _Affective computing_.  MIT press.
  * Puglisi and Ackerman (2016a) Becca Puglisi and Angela Ackerman. 2016a.  _The Rural Setting Thesaurus: A Writer’s Guide to Personal and Natural Places_ , volume 4.  JADD Publishing.
  * Puglisi and Ackerman (2016b) Becca Puglisi and Angela Ackerman. 2016b.  _The Urban Setting Thesaurus: A Writer’s Guide to City Spaces_ , volume 5.  JADD Publishing.
  * Puglisi and Ackerman (2019) Becca Puglisi and Angela Ackerman. 2019.  _The emotion thesaurus: A writer’s guide to character expression_ , volume 1.  JADD Publishing.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Reniers et al. (2011) Renate LEP Reniers, Rhiannon Corcoran, Richard Drake, Nick M Shryane, and Birgit A Völlm. 2011.  The qcae: A questionnaire of cognitive and affective empathy.  _Journal of personality assessment_ , 93(1):84–95.
  * Resnik (1967) Michael David Resnik. 1967.  The context principle in frege’s philosophy.  _Philosophy and Phenomenological Research_ , 27(3):356–365.
  * Sap et al. (2022) Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022.  Neural theory-of-mind? on the limits of social intelligence in large lms.  _arXiv preprint arXiv:2210.13312_.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019.  Socialiqa: Commonsense reasoning about social interactions.  _arXiv preprint arXiv:1904.09728_.
  * Schindler et al. (2008) Konrad Schindler, Luc Van Gool, and Beatrice De Gelder. 2008.  Recognizing emotions expressed by body pose: A biologically inspired neural model.  _Neural networks_ , 21(9):1238–1246.
  * Smaira et al. (2020) Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. 2020.  A short note on the kinetics-700-2020 human action dataset.  _arXiv preprint arXiv:2010.10864_.
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_.
  * Stefanini et al. (2022) Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. 2022.  From show to tell: A survey on deep learning-based image captioning.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):539–559.
  * Tanaka et al. (2010) Akihiro Tanaka, Ai Koizumi, Hisato Imai, Saori Hiramatsu, Eriko Hiramoto, and Beatrice De Gelder. 2010.  I feel your voice: Cultural differences in the multisensory perception of emotion.  _Psychological science_ , 21(9):1259–1262.
  * Team et al. (2023) MosaicML NLP Team et al. 2023.  Introducing mpt-30b: Raising the bar for open-source foundation models.
  * Thuseethan et al. (2021) Selvarajah Thuseethan, Sutharshan Rajasegarar, and John Yearwood. 2021.  Boosting emotion recognition in context using non-target subject information.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–7. IEEE.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Veltmeijer et al. (2021) Emmeke A Veltmeijer, Charlotte Gerritsen, and Koen V Hindriks. 2021.  Automatic emotion recognition for groups: a review.  _IEEE Transactions on Affective Computing_ , 14(1):89–107.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3156–3164.
  * Wang et al. (2022) Zili Wang, Lingjie Lao, Xiaoya Zhang, Yong Li, Tong Zhang, and Zhen Cui. 2022.  Context-dependent emotion recognition.  _Journal of Visual Communication and Image Representation_ , 89:103679.
  * Yang et al. (2023) Vera Yang, Archita Srivastava, Yasaman Etesam, Chuxuan Zhang, and Angelica Lim. 2023.  Contextual emotion estimation from image captions.  _arXiv preprint arXiv:2309.13136_.
  * Yang et al. (2021) Xiaocui Yang, Shi Feng, Yifei Zhang, and Daling Wang. 2021.  Multimodal sentiment detection based on multi-channel graph neural networks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 328–339.
  * You et al. (2016) Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2016.  Building a large scale dataset for image emotion recognition: The fine print and the benchmark.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30.
  * Zhao et al. (2016) Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, and Guiguang Ding. 2016.  Continuous probability distribution prediction of image emotions via multitask shared sparse regression.  _IEEE transactions on multimedia_ , 19(3):632–645.
  * Zhao et al. (2021) Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjoern W Schuller, and Kurt Keutzer. 2021.  Affective image content analysis: Two decades review and new perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(10):6729–6751.
  * Zhu et al. (2016) Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016.  Visual7w: Grounded question answering in images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4995–5004.

##  Appendix A Appendix

Table 7: The Emotion Thesaurus Puglisi and Ackerman (2019) lists physical signals for numerous emotions; we included only the signals from the emotions which were synonyms of the EMOTIC labels listed here.  EMOTIC Labels | The Emotion Thesaurus
---|---
affection | love, adoration
anger | anger, resentment, rage
annoyance | annoyance, irritation, impatience, frustration
anticipation | anticipation
aversion | disgust, hatred, reluctance
confidence | confidence, certainty, pride
disapproval | contempt, scorn
disconnection | indifference, boredom
disquietment | nervousness, worry, anxiety
doubt/confusion | confusion, doubt, skepticism
embarrassment | embarrassment, guilt, shame
engagement | curiosity
esteem | gratitude, admiration
excitement | excitement
fatigue | exhaustion, lethargy
fear | fear, suspicious, terror, horror
happiness | happiness, amusement, joy
pain | pain
peace | peacefulness, satisfaction, relaxation
pleasure | pleased
sadness | sadness, disappointment, disappointed, discouragement
sensitivity | vulnerability
suffering | anguish, depression, stress, hurt
surprise | surprise/shock
sympathy | sympathy
yearning | envy, jealousy, desire, lust
Table 8: Example Listing of Physical Signals from Puglisi and Ackerman (2019). Full list is not included due to copyright. Physical Signal Description |
---|---
Has a belly laugh | Has a clenched jaw
Has a bent back | Has a collapsed body posture
Has a bent neck | Has a confrontational stance
Has a bitter smile | Has a corded neck
Has a body that curls in on itself | Has a death grip on a purse
Has a bowed head | Has a head tilt
Has a deliberate eyebrow raise | Has a deliberate lowering of the head
Has a distant or empty stare | Has a downturned mouth
Has a downward gaze | Has a drunken behavior
Has a exposed the neck | Has a fight response
Has a flat look | Has a flush visible in the cheeks
Has a flushed appearance | Has a gaunt appearance
Has a grimace or pained look | Has a hand casually anchored on the hip
Has a hand flap that dismisses the person or their idea | Has a hand fluttering to the lips or neck
Has a hand pressing against the throat or breastbone | Has a hanging head
Has a hard, distinctive jaw line | Has a harried appearance
Has a head that tips back, exposing the neck | Has a high chin
Has a hunched posture | Has a mouth that curls with dislike, sneering
Has a pained expression | Has a pained gaze
Has a pained or watery gaze | Has a pained stare
Has a pinched expression | Has a pinched mouth
Has a pouty bottom lip | Has a red face and neck
Has a relaxed physique | Has a runny nose
Has a set jaw | Has a shiny face
Has a silly grin | Has a slack expression
Has a smile that appears tight | Has a stiff neck
Has a stony expression | Has a straight posture
Has a tension-filled expression | Has a thrust-out chest
Has a tilted-back head | Has a two-fingered salute
Has a welcoming stance | Has a wide grin
Has a wide, open stance | Has a wrinkled brow
Has a wrinkled nose | Has a yearning look
Has aggression | Has an arm hooked over the back of a chair
Has an expression that appears pained | Has an intense, fevered stare
Has an inward focus | Has an open mouth
Has an overall lifting of the facial countenance | Has an overall visage that glows
Has an unclean home, room, or office space | Has an unkempt appearance
Has an upturned face | Has angry tears
Has arms crossing in front of the chest |
Figure 3: Annotation interface for human annotators. The annotator follows the
annotation procedures as follows: 1. indicate the bounding box for the subject
they are annotating 2. select the perceived age, and social identity of the
subject 3. Go over the lists of physical signals for eyes, mouth, face, hands,
torso, and feet, and check the ones they see fit 4. Choose the correct
bounding box of a subject who has interaction with the subject of interest 5.
select the perceived sex, and describe the inferred relation and interaction
between the two subjects 6. Describe the subject’s physical surroundings. The
captions generated are displayed for the annotator to review easily.


11institutetext: Institute of Physics, EÃtvÃs University, PÃ¡zmÃ¡ny PÃter
SÃtÃ¡ny 1/A, H-1117 Budapest, Hungary 22institutetext: Wigner Research Centre
for Physics, Konkoly-Thege MiklÃs Ãt 29-33, H-1121 Budapest, Hungary.

# Estimating Microscopic Nuclear Data by Compact Star Observations

BalÃ¡zs Endre Szigeti 1122<EMAIL_ADDRESS>Gergely GÃ¡bor BarnafÃldi
22 PÃter PÃsfay 22 Antal JakovÃ¡c 22

###### Abstract

We studied recent observation data of pulsar masses and radii of PSR
J0740$+$6620, PSR J0348$+$0432, and PSR J1614$-$2230 from different
measurements, based on the extended version of $\sigma$-$\omega$ model.
Throughout our analysis, we assumed that these pulsars are maximal-mass
compact stars, thus we applied the core approximation. Based on the linear
relation between the microscopic and macroscopic parameters of compact stars
evaluated by our model, we estimated the average Landau mass
$m_{L}=752.46^{+49.1}_{-42.5}$ MeV and compressibility
$K=261.7^{+57.2}_{-28.0}$ MeV.

##  1 Introduction

Compact star observables are depend on the internal characteristics of the
dense nuclear matter. Modification of the microscopic nuclear model and its
parameters can vary the measureable macroscopic properties of a pulsar as
well. In our analysis we try to approach the problem from the inverse
direction and attempt to estimate the properties of the nuclear matter from
compact star observables, which is made more challenging by the masquarade
problem. It states that even large variation of the nuclear matter parameters
and by exchanging of the interaction terms result in stars with very similar
observable properties.

The effect of various interaction terms and the variation of the nuclear
parameter values in the Lagrangian has been presented so far for the case of
the extended $\sigma$-$\omega$ model Posfay:2020 . Linear dependence of the
mass and radius parameters on the Landau mass, $m_{L}$, compressibility, $K$,
and nuclear asymmetry, $a_{sym}$ was observed and tested for maximal mass
(MMS) neutron star scenarios. Ordering in the strengths of these properties
variation has been also obtained in general for the maximum mass star (MMS):
$\Delta M_{max}(\delta m_{L})\overset{10\times}{>}\Delta M_{max}(\delta K)$.
Landau effective mass values we already determined by former analysis
Alwarez:2020 ; Barnafoldi:2020 . We would like to investigate the
compressibilty $K$ by using recent maximal-mass pulsar observation data from
PSR J0740$+$6620 J0740 ; XMNS ; NICER , PSR J0348$+$0432 J0348 , and PSR
J1614$-$2230 J1614 . We estimate the mean $K$ and $m_{L}$ value including the
uncertainties originating from the theory and data. The model and the analysis
method was described in detail in Ref. Alwarez:2020 ; Barnafoldi:2020 .

##  2 The Model and the Equation of State

One of the most commonly used model to describe the interior structure of the
compact star is the $\sigma$-$\omega$ model, which can be naturally extended
by further interactions Posfay:2020 ; Posfay:2019 . This model describes
protons, electrons, and neutrons in $\beta$-equilibrium and approximates the
nuclear force by introducing the $\sigma$, $\omega$, and $\rho$ meson with
higher-order self-interaction terms for the $\sigma$ meson. In our study we
would like to focus on the connection between macro- and microscopical
parameters, which led us to estimate nuclear parameters with high precision
from pulsar data based on this model. The Lagrangian of the corresponding to
the $\sigma$-$\omega$ model is,

$\displaystyle\mathcal{L}$ $\displaystyle=$
$\displaystyle\overline{\Psi}\left(i\not{\partial}-m_{N}+g_{\sigma}\sigma-
g_{\omega}\not{\omega}+g_{\rho}\not{\rho}^{a}\tau_{a}\right)\Psi+\overline{\Psi}_{e}\left(i\not{\partial}-m_{e}\right)\Psi_{e}-\lambda_{3}\sigma^{3}+\lambda_{4}\sigma^{4}$
$\displaystyle+\frac{1}{2}\,\sigma\left(\partial^{2}-m_{\sigma}^{2}\right)\sigma-\frac{1}{4}\,\omega_{\mu\nu}\omega^{\mu\nu}+\frac{1}{2}m_{\omega}^{2}\,\omega^{\mu}\omega_{\mu}-\frac{1}{4}\rho_{\mu\nu}^{a}\,\rho^{\mu\nu\,a}+\frac{1}{2}m_{\rho}^{2}\,\rho_{\mu}^{a}\,\rho^{\mu\,a}\
,$

where $\Psi=(\Psi_{n},\Psi_{p})$ is the vector of proton and neutron fields,
$m_{N}$, $m_{\sigma}$, $m_{\omega}$, $m_{\rho}$ are the masses of the nucleons
and $\sigma$ is the scalar-, and $\omega$, $\rho$ are (iso-)vector mesons
respectively. Moreover $g_{\sigma}$, $g_{\omega}$, and $g_{\rho}$ are the
Yukawa couplings corresponding to the nucleon-meson interactions. The kinetic
terms corresponding to the $\omega$ and $\rho$ meson can be written in the
following forms,

$\omega_{\mu\nu}=\partial_{\mu}\omega_{\nu}-\partial_{\nu}\omega_{\mu}\ \
\textrm{and }\ \
\rho_{\mu\nu}^{a}=\partial_{\mu}\rho_{\nu}^{a}-\partial_{\nu}\rho_{\mu}^{a}+g_{\rho}\epsilon^{abc}\rho_{\mu}^{b}\rho_{\nu}^{c}.\,$
(2)

We are taking the mean-field approximation at zero temperature and finite
chemical potential of this model. Although in the Lagrangian we can consider
the fluctuations of the $\omega$ and $\rho$ vector meson fields, but because
they are relative heavy their contribution can be neglected in the loop-
integrals, only the $\omega_{0}=\omega$ and $\rho_{0}^{3}=\rho$ components
gives non-zero value, due to symmetrical reasons but kinetic terms are
disappeared. From this point the free energy corresponding to the model can be
calculated as it is described in for example in Ref. jakovac2015resummation :

$\displaystyle f_{T}$ $\displaystyle=$ $\displaystyle
f_{F}\left(m_{N}-g_{\sigma}\sigma,\mu_{p}-g_{\omega}\omega+g_{\rho}\rho\right)+f_{F}\left(m_{N}-g_{\sigma}\sigma,\mu_{n}-g_{\omega}\omega-
g_{\rho}\rho\right)+f_{F}\left(m_{e},\mu_{e}\right)$ (3) $\displaystyle+\hskip
2.84544pt\frac{1}{2}m_{\sigma}^{2}\sigma^{2}+U_{i}(\sigma)-\frac{1}{2}m_{\omega}^{2}\omega^{2}-\frac{1}{2}m_{\rho}^{2}\rho^{2}\,,$

where $\mu_{p}$, $\mu_{n}$ and $\mu_{e}$ are the proton, neutron, and electron
chemical potential respectively. The $f_{F}$ term describes the free energy
contribution corresponding to one fermionic degree of freedom Posfay:2020 .
For the fit of the free parameters of this model the nuclear saturation data
were used meng2016relativistic , except for the Landau mass. Then after
getting the optimal Landau mass value, we estimate the values for the
compressibility as well. These two parameters are kept free and determined by
comparing the mass radius diagrams corresponding to different values of the
Landau mass to neutron star observations. The asymmetry energy plays
negligible role on the mass and radius values of the maximal mass compact
stars as it was pointed out in Ref. Posfay:2020 . For the general relativistic
description of the compact stars we assumed the usual static picture in
spherically symmetric space time. We calculated the mass-radius diagram by the
well-known Tolman – Oppenheimer – Volkoff equations (TOV). To focus our
investigation on the effect of the nuclear matter we integrated the TOV
equations, and we employed a stopping condition for the integration based on
Ref. Posfay:2020 : $p(r=R^{\prime})=p_{0}$. Here, $p_{0}$ is chosen such a way
that the integration stops at the core of the neutron star, so the integration
does not take into account the effect of the crust and the low density EoS. To
get a good approximation for $p_{0}$ we used the well known BPS nuclear
equation of state which is used to describe the crust of neutron stars BPS .
We used the highest pressure value for $p_{0}$ where the BPS EoS can still be
considered valid.

The calculated $R^{\prime}$ in this case corresponds to the radius of the
neutron star core. The mass and radius data calculated this generally can be
considered a conservative approximation of the neutron star parameters,
however in the case of the maximum mass star the deviations from the normal
case are insignificant Posfay:2020 .

##  3 Results

Assuming maximal mass stars, we found linear dependence of both observables,
the mass and radius of the maximal mass star, $M_{maxM}$ and $R_{maxM}$ on the
$m_{L}$, where we used the saturated nuclear matter for the compressibility
and asymmetry energy. The functions were fitted independently with $0.8\%$ and
$17\%$ theoretical uncertainty, respectively, following Ref Posfay:2020 :

$\displaystyle M_{maxM}[\textrm{M}_{\odot}]=5.418-0.0043\,m_{L}[\textrm{MeV}]\
,$ (4) $\displaystyle R_{maxM}[\textrm{km}]=19.04-0.0104\,m_{L}[\textrm{MeV}]\
.$ (5)

After we fixed the Landau mass by the MMS observation we can also obtain a
one-parameter linear equation between the compressibility, $K$ and the mass
and radius of the MMS compact star with $\lesssim 2\%$ and $\lesssim 14\%$
theoretical uncertainty,

$\displaystyle M_{maxM}[\textrm{M}_{\odot}]=1.940+0.000880\,K\,[\textrm{MeV}]\
,$ (6) $\displaystyle R_{maxM}[\textrm{km}]=9.248+0.00718\,K\,[\textrm{MeV}]\
.$ (7)

One can see that the $K$-equations (6) and (7) have positive gradient which
means that increasing compressibility makes an MMS compact star more larger
and massive. Using these derived equations and pulsar observation data we
obtained the optimal Landau masses, and compressibility values for each
compact object in Table 1. We also determined the average of Landau masses
$m_{L}=752.46^{+49.1}_{-42.5}$ MeV and mean compressibility
$K=261.7^{+57.2}_{-28.0}$ MeV, which agrees well with the saturated nuclear
matter data and with our recent works Alwarez:2020 ; Barnafoldi:2020 . In
contrary, calculations of $R_{maxM}$ by eqs. (5) and (7) present large
uncertainties, especially in case of radius data from Ref. NICER , thus core
approximation might not be valid, due to the softer EoS and its relevant
contribution to the crust. This result in a $\gtrsim 1$ km larger radius.

Table 1: The Landau mass, $m_{L}$ and compressiblity, $K$ values evaluated via eqs. (4)-(7) from measured pulsar masses and radii data denoted by â$\ast$â, and assuming that these are maximal-mass neutron stars. Ref. | Pulsar |  $M_{maxM}$[M⊙] |  $m_{L}$[MeV] |  $K$[MeV] |  $R_{maxM}$[km]
---|---|---|---|---|---
J0740  | PSR J0740+6620 | 2.17${}^{+0.11\ \ast}_{-0.10}$ | 748.39${}^{+63.3}_{-57.2}$ | 351.8${}^{+115\ \ }_{-84.5}$ | 11.25${}^{+1.06}_{-1.04}$
J0348  | PSR J0348+0432 | 2.01${}^{+0.04\ \ast}_{-0.04}$ | 785.25${}^{+20.0}_{-20.3}$ | 206.4${}^{+42.7\ \ }_{-20.5}$ | 10.87${}^{+0.82}_{-0.80}$
J1614  | PSR J1614-2230 | 1.97${}^{+0.04\ \ast}_{-0.04}$ | 794.47${}^{+20.1}_{-20.4}$ | 170.0${}^{+15.5\ \ }_{-20.9}$ | 10.77${}^{+0.82}_{-0.80}$
NICER  | PSR J0740+6620 | 2.07${}^{+0.07\ \ast}_{-0.07}$ | 778.14${}^{+15.3}_{-15.5}$ | 278.2${}^{+60.9\ \ }_{-60.8}$ | 11.01${}^{+0.46}_{-0.47}$
NICER  | PSR J0740+6620 | 2.64${}^{+1.98}_{-0.65}$ | 639.42${}^{+159}_{-125}$ | 277.3${}^{+257\ \ }_{-178}$ |  12.39${}^{+1.30\ \ast}_{-0.98}$
XMNS  | PSR J0740+6620 | 2.08${}^{+0.07\ \ast}_{-0.01}$ | 769.12${}^{+16.9}_{-16.9}$ | 285.1${}^{+54.8\ \ }_{-54.8}$ | 11.06${}^{+0.41}_{-0.41}$

Figure 1: Left panel: mass of the maximal mass star, $M_{maxM}(m_{L},K)$.
Right panel: radius of the maximal mass star, $R_{maxM}(m_{L},K)$. Curves at
both panels are as function of $K$ at various physical $m_{L}$ values, denoted
as color lines. Comparison to pulsar data of PSR J0740$+$6620 J0740 ; XMNS ;
NICER , PSR J0348$+$0432 J0348 , and PSR J1614$-$2230 J1614  are also plotted,
including theoretical uncertainties and data errors.

On the Fig. 1 we plotted the projections of the $M_{maxM}(m_{L},K)$ and
$R_{maxM}(m_{L},K)$ curves, where the $K$-dependent one-parameter linear
equations are determined for various $m_{L}$ values. Pulsar mass and radii
data from Table 1. are also plotted with color markers, where the error bars
include both theoretical uncertainties from the phenomenological approach and
errors from the astrophysical observation. Our results agree well with our
previous results Posfay:2020 ; Barnafoldi:2020  and also with a Bayesian
analysis carried out on this extended $\sigma$-$\omega$ model Alwarez:2020 ;
Barnafoldi:2020 .

##  4 Summary

Macroscopical compact star observation parameters and microscopical superdenes
nuclear matter properties has been formulated in Refs. Posfay:2020 ;
Alwarez:2020 ; Barnafoldi:2020 . This has been validated in the liu of the
recent NICER NICER  and NICER+XMM-Newton XMNS  data. We find that the concept
of MMS star is valid, until the core approximation stands. Microscopic nuclear
parameter values: mean Landau mass $m_{L}=752.46^{+49.1}_{-42.5}$ MeV and
compressibility $K=261.7^{+57.2}_{-28.0}$ MeV were given.

##  5 Acknowledgement

Authors acknowledge for NKFIH (OTKA) grants No. K123815, K135515, PHAROS
(CA16214) cost action, and for the computational resources by the Wigner GPU
Laboratory.

## References

  * (1) P. Pósfay, G. G. Barnaföldi and A. Jakovác, arXiv:2004.08230 [astro-ph.HE].
  * (2) D. Alvarez-Castillo et al, Eur. Phys. J. ST 229 no.22-23, 3615-3628 (2020)
  * (3) G. G. Barnaföldi et al, Eur. Phys. J. ST 229 no.22-23, 3605-3614 (2020)
  * (4) H. Cromartie et al, Nat. Astron. 4, 72-76 (2019)
  * (5) M.C. Miller et al, arXiv:2105.06979 [astro-ph.HE] (2021)
  * (6) T.E. Riley et al, arXiv:2105.06980 [astro-ph.HE] (2021)
  * (7) J. Antoniadis et al, Science 340, 6131 (2013)
  * (8) Demorest, P.et al,Nature 467, 1081-1083 (2010).
  * (9) P. PÃsfay, G. G. BarnafÃldi, and A. JakovÃ¡c, Universe 5, 153 (2019).
  * (10) A. JakovÃ¡c and A. PatkÃs, Resummation and Renormalization in Effective Theories of Particle Physics, Lecture Notes in Physics (Springer International Publishing, 2015)
  * (11) J. Meng, Relativistic Density Functional for Nuclear Structure, International Review of Nuclear Physics (World Scientific Publishing Company, 2016),
  * (12) G. Baym, C. Pethick, and P. Sutherland, Astrophysical Journal, 170, p.299 (1971)