§ INTRODUCTION

Figurative language is the use of words or phrases in a way that deviates from their literal meaning, aiming to evoke specific concepts or imagery within one's imagination[<https://www.ldoceonline.com/dictionary/figurative>]. Figurative language consists of different types <cit.>, such as metaphors, which use comparisons to describe something differently (e.g., the road ahead is a long and winding journey); idioms, which are common phrases that have alternate meanings (e.g., to beat around the bush); similes, which use like or as to compare two things (e.g., as light as a feather); and personification, which gives human qualities to objects or animals (e.g., the leaves danced in the wind).

Within the software engineering (SE) community, professionals often employ various distinctive figurative expressions that are not commonly used in everyday discourse. For instance, developers utilize the metaphorical term anti-pattern to communicate the idea of a recurring problem that should be avoided <cit.>. Idioms, another frequently employed form of figurative expression, play a crucial role in SE communication by succinctly and colloquially conveying common ideas or concepts. An example of this is when developers describe poorly written code as spaghetti code, implying that it is convoluted and challenging to comprehend <cit.>.

Just as humans use phrases like boil with anger or a breath of fresh air for relief <cit.>, developers might say a thorn in my side[<https://github.com/chipsalliance/chisel/pull/3352#issuecomment-1593479230>] to express persistent annoyance or difficulty with an API or a feature. Use of pejorative terms like garbage code [<https://github.com/adamit24/countdownClass/issues/1>], Frankencode [<https://github.com/drupal-code-builder/drupal-code-builder/issues/165>] can be indicators of severe negative emotions, leading to toxic discussions.
Therefore, understanding the use of figurative language in software development discourse can help detect the use of offensive language <cit.> and provide valuable insights into the overall health of a software project <cit.>.
Developers also use figurative expressions to indicate the impact and severity of a bug. For instance, while expressions like a ticking time bomb [<https://github.com/godotengine/godot/issues/77480#issue-1726201628>], suggests significant future problems,  showstopper [<https://github.com/conwid/VSCleanBin/issues/2#issuecomment-571094076>] and critical roadblock [<https://github.com/canjs/canjs/pull/3286>], emphasize the urgency of addressing the bug at the present.
A recent blog article noted how an improved understanding of software engineering metaphors would mitigate the risks of misinterpretations, fostering more precise and effective communication within the software development community[<https://www.leadingagile.com/2018/11/metaphorically-speakingthe-history-of-communication-in-software/>], while a recent study showed figurative language like Humor has positive effect on developer engagement <cit.>.
Recent research studies have also highlighted that flaws in SE emotion and sentiment detection tools stem from the use of figurative language <cit.>. On a Stack Overflow and GitHub dataset, Novielli et al. <cit.> found that 9% of the errors in sentiment analysis were due to figurative language, noting that it poses an open challenge for sentiment detection in software engineering.

Despite its potential impact on the performance of automatic tools focused on understanding SE text, there have been very limited studies on analyzing figurative language in SE, e.g., there have been some studies on SE synonyms <cit.> and programming language-specific idioms <cit.>.
In this paper, we aim to go beyond the synonyms and explore the broader landscape of figurative language in SE. We aim to shed light on or analyze the use of figurative language (specifically, metaphors and idioms) in SE communication channels and contribute to the understanding of how recently proposed language models that target software-related text can be made to recognize figurative expressions.

Large Language Models (LLMs), such as BERT <cit.> and RoBERTa <cit.>, have recently demonstrated state-of-the-art results on a variety of software engineering tasks, e.g., code completion, code review, bug localization, sentiment analysis, toxicity detection <cit.>. While LLMs are not explicitly designed to detect figurative languages like metaphors and idioms, they can acquire this ability through training on large datasets such as Wikipedia and Stack Overflow <cit.>. This capability is particularly beneficial in the software engineering context, as it enables a more nuanced and accurate analysis of developer communications. Without this ability, an LLM may misinterpret or misclassify text, leading to erroneous results. For instance, if an LLM cannot recognize the idiom edge case, it may interpret the phrase literally and erroneously categorize the text as being related to a specific type of physical boundary instead of grasping its figurative meaning of a rare or unusual scenario.

Through this study, we will examine the relevance of figurative language in GitHub communication channels, the ability of LLMs to detect figurative language in the SE context, and the impact of figurative language on affect analysis and bug report priority detection. By gaining a deeper understanding of the role and effects of figurative language in SE, we aim to contribute to the development of more effective and accurate NLP-based systems for SE tasks, specifically in automated recognition of developer emotions and incivility on GitHub, and bug report priority detection. We focus on answering the following three research questions:

RQ1: How well can existing LLMs interpret figurative language (i.e., metaphors and idioms) used in software engineering?

To answer this RQ, we collect a set of 2000 sentences containing figurative language and create rephrased sentences, i.e., sentences with similar meanings but without figurative expressions. We also create altered sentences that share as many words as the original sentence but convey different meanings, e.g., using metaphors in their literal sense or using idioms in a different context other than software engineering.
This procedure of creating, so called, entailed and non-entailed text from premise text has been widely used in NLP <cit.>. Using this data triple of original, rephrased, and altered meaning sentences,
we investigate whether LLMs can recognize the semantics of figurative sentences by computing how often the models identified the semantic dissimilarity of the rephrased sentence with the altered sentence. Our results suggest that LLMs have a limited ability to interpret figurative language, with higher performance for general figurative expressions than software engineering-specific ones.

RQ2: Can the performance of software engineering-specific affective analysis be improved by a better insight into figurative language?

Affect expressions are the means to convey emotions, feelings, and attitudes to others <cit.>. For some time now, researchers have been exploring automatic affect analysis, which encompasses tasks such as emotion analysis, sentiment analysis, and incivility analysis. To answer RQ2, we fine-tune several LLMs using contrastive learning <cit.> with our dataset of figurative language in order to improve their ability to interpret figurative language. We then compare the performance of the fine-tuned LLMs to the original models of two publicly available affect datasets: an emotion dataset curated from GitHub, and an incivility dataset curated from GitHub. Our results indicate that fine-tuned LLMs perform better in both cases.

RQ3: Can a better understanding of figurative language enhance software engineering automation where affect plays a role?

A number of research tasks in SE indirectly involve affective natural language text, e.g., app review analysis, opinion mining <cit.>. Specifically, in this RQ we investigate how a better understanding of figurative language can impact bug report priority detection, which is a significant area of interest in open-source software research <cit.>. Umer et al. observed that emotions influence bug report priority detection <cit.>. To address this problem, recently researchers have employed Language Models (LLMs) <cit.>. In this study, we explore LLMs fine-tuned with contrastive learning using our figurative language dataset, similar to the approach in RQ2, and conducted experiments on the publicly available Bugzilla dataset[<https://bugs.eclipse.org/bugs/>].
Our results indicate that fine-tuning with our figurative language dataset improves bug report priority detection.

We publish the annotation instructions, annotated dataset, and source code to facilitate the replication of our study at <https://github.com/vcu-swim-lab/SE-Figurative-Language>.

§ DATASET

To conduct our study, we curate a dataset of developer communications containing figurative language. Towards that goal, we first collect data from GitHub issues and pull requests and identify the occurrences of idioms and metaphors. To inquire whether language models understand figurative language, we manually rephrase the original sentences containing figurative language to generate: 1) sentences that are similar in meaning to the original but do not contain idioms or metaphors; and 2) sentences that contain similar words as the original sentences but are semantically dissimilar, i.e., have a different meaning.
In this section, we detail each step involved in constructing our dataset.

§.§ Data Collection

We selected nine popular GitHub repositories, each with a minimum of 50k stars: skylot/jadx, laravel/laravel, microsoft/PowerToys, rails/rails, redis/redis, facebook/react, tensorflow/tensorflow, huggingface/transformers, and microsoft/vscode. We collected 10k comments from each repository (5k PR comments and 5k issue comments) between February 2022 and May 2023.
We split the comments into sentences using NLTK[<https://www.nltk.org/>] and filtered out sentences with fewer than 5 words, resulting in a total of 202k sentences.

One of our study's end goals is to examine figurative language's impact on affective expressions in a software engineering context. Previous research has shown that most comments on GitHub are neutral, lacking any detectable emotions or sentiments <cit.>. Therefore, we excluded neutral sentences by using a software engineering-specific sentiment analysis tool <cit.>.

In addition, to avoid including sentences that do not contain any figurative expressions, we applied a popular metaphor detection <cit.> and an idiom detection tool <cit.> to identify candidate metaphors and idioms in each sentence. This model-in-the-loop approach is popular in Natural Language Inference (NLI) research, e.g., figurative language interpretation <cit.>, as it maximizes the value of annotation effort, which requires tedious human labor. We discarded sentences that do not contain any candidate idioms or metaphors. We randomly selected 1000 sentences containing metaphors from the remaining sentences. We also randomly chose 1000 sentences containing idioms (different from the metaphor set). This process resulted in a dataset of 2000 sentences.

§.§ Data Annotation

First, we recruited four annotators (two graduate students and two senior undergraduate students) who were each given 500 sentences to annotate (250 with metaphors and 250 with idioms). Due to the nature of the task and difficulties with crowd-sourcing <cit.>, we opted for a small number of annotators
that are native speakers/professionally fluent in English with a strong computer science background.
Along with the 2000 sentences in total, the annotators were provided with a set of candidate figurative expressions marked by the above-mentioned tools. We instructed them to: 1) verify the candidates as metaphors or idioms and judge whether each metaphor or idiom is specific to software engineering or general purpose; and 2) create rephrased sentences from the original. We also held a short training session in which we reviewed the annotation process for a few representative examples with each annotator. Below, we describe these data annotation steps in detail (see also Figure <ref>).

§.§.§ Verifying Figurative Expressions
For verifying metaphors and idioms, we followed best practices from existing literature. More specifically, to verify the metaphors we asked the annotators to carefully read the Metaphor Identification Procedure (MIP) guideline by the Pragglejaz Group <cit.>. The MIP guideline is a well-known procedure for identifying metaphors. Based on the guideline, the annotators marked the correct metaphoric expressions from the candidate set. For example, the annotators confirmed that nasty bug is a valid metaphor for a difficult fault in the sentence, Otherwise, this could give us a nasty bug.

We noted in the annotation instructions that most metaphors are conventional, i.e., metaphors that are often used in everyday language <cit.>. For example, in the following sentence: I see your point, see and point both are metaphors <cit.>. Often such cases can be observed in software engineering communication. For instance, pinging in the following sentence is a metaphor: Hi @[USER], thanks for pinging me on this issue. Here, pinging is a colloquial way of saying contacting someone, while the literal meaning of pinging comes from computer networking terminology[<https://ftp.arl.army.mil/ mike/ping.html>].

For verifying idioms, we followed the guideline provided by Stowe et al. <cit.>, which asked the annotators to look up idioms in popular dictionaries (such as the Oxford English Dictionary[<https://www.oxfordlearnersdictionaries.com/>], the Webster Dictionary[<https://www.merriam-webster.com/>], and the Longman Dictionary of Contemporary English[<https://www.ldoceonline.com/>] and popular search engines (e.g., Google).
We instructed the annotators to consider an expression as likely to be an idiom if its dictionary definition is: 1) applicable in the context; and 2) a good syntactic fit in the same environment. For example, in the sentence, I will also be keeping an eye on you, keeping an eye is an idiom which means to watch someone or something or stay informed about the person's behavior, especially to keep someone out of trouble.[<https://dictionary.cambridge.org/>] Conversely, when the meaning of the candidate idiomatic expression is literal in the context of the sentence and the dictionary definition is not applicable, it is likely not to be an idiom. For example, in the sentence, It was cold, so cold in the jeep that it was with difficulty that Alexei kept his eyes open, kept his eyes open is not an idiom.
Since software-specific words have distinct meanings from conventional terms (e.g., bug, issue, error, function), we supplied annotators with established software engineering glossary terms from the FDA[<https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/inspection-guides/glossary-computer-system-software-development-terminology-895>] and Google[<https://developers.google.com/machine-learning/glossary>].

Once annotators verified the candidate set, we asked them to mark whether the figurative expressions were software engineering-specific or general-purpose. The annotators identified 752 sentences with metaphors and 909 with idioms, totaling 1661 sentences. The remaining 339 sentences did not contain any figurative words. These 1661 sentences had a total of 1741 unique figurative expressions, with 445 being SE-specific and 1296 general.

Figurative language annotation procedure.

§.§.§ Rephrasing Sentences

The process of rephrasing sentences was divided into two phases: creating semantically-equivalent rephrased sentences and constructing altered-meaning sentences. We refer to the semantically equivalent rephrased sentences as  (EMS) throughout the paper. These sentences retain the original meaning of the sentence, but the figurative expressions are replaced with literal terms. We refer to the altered-meaning sentences as   (DMS). These sentences are modified so that they significantly differ in meaning from the original sentences.

a) EMS Construction: The annotators were tasked with rephrasing each sentence on their list, i.e., removing the (verified) figurative expressions while maintaining the original semantics of the sentence as much as possible. In other words, the replaced figurative expression should entail its literal counterpart. For example, in the sentence, [USER] Thanks for your help, what you said may be a hidden bug., the figurative expression hidden bug is replaced with unseen error resulting in the EMS: [USER] Thanks for your help, what you said may be an unseen error. This approach is inspired by previous research by Stowe et al. on figurative language in NLP <cit.>. It is worth noting that for EMS we did not employ multiple annotators to annotate the same set or calculate inter-annotator agreement as Stowe et al. found that this method does not yield significantly different quality compared to the conventional approach <cit.>.

b) DMS Construction: s are variations of metaphorical or idiomatic sentences that convey a different meaning than the original sentence and do not entail it <cit.>. Two strategies were employed to construct DMS: a) using figurative expressions in a literal sense; and b) replacing the figurative expressions and their context with different words. These strategies are inspired by previous research on figurative language in natural language processing <cit.>. We also apply a model-in-the-loop approach for DMS generation <cit.>. More specifically, we first generated four candidate DMSs for each sentence, two each for each strategy using GPT-4 <cit.> API (`gpt-4'[<https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo/>]), and, second, we recruited human annotators to select the best-generated candidate (or to create one of their own if none is available).

Candidate DMS Generation. ChatGPT <cit.> has shown promising results in data annotation tasks, including text generation, in some cases outperforming human crowd-workers <cit.>. Following recent literature, we create two GPT-4 prompts for the two different strategies for DMS generation <cit.>. The prompts were carefully devised by using the existing literature on this topic <cit.>.
The prompts for generating DMS are as follows:

Generating DMS by using the figurative language in a literal manner:
You are reading GitHub comments with figurative expressions. Your task is to generate 2 examples by using the given figurative expressions in a literal manner to construct different sentences. Do not replace them. Add/change new contexts if necessary. The new sentence must have a completely different meaning than the original. You must keep the semantic order of the original sentences as much as possible. Don't explain your answer.

Original Sentence:<insert utterance>.

Figurative expressions: <insert figurative expressions>

Generating DMS by replacing the figurative language:
You are reading GitHub comments with figurative expressions. Your task is to generate 2 examples by replacing given figurative expressions to construct different sentences. The new sentence must have a completely different meaning than the original. You are only allowed to change the figurative expression and its context. You must keep the semantic order of the original sentences as much as possible. Don't explain your answer.

Original Sentence:<insert utterance>.

Figurative expressions: <insert figuration expressions>

DMS Selection. Two additional annotators (one of the authors and one senior undergraduate student) were responsible for the candidate selection of the DMS. We provided the annotators the original sentence, the figurative expressions, and the list of candidate DMSs with the following instructions: You will be provided with 4 candidate sentences, two of which come from Type 1 and two come from Type 2. Choose the best 1 out of the 4 candidates, with a preference towards choosing from Type 1. If none of these 4 are good candidates, write None. When choosing, try to choose a sentence that has 1) similar semantic order to the original sentence, and 2) a different meaning than the original sentence.

We instructed the annotators to write their own DMS when their selection is None.
Once they completed an annotation pass over the entire dataset, the two annotators met in person in order to discuss the 310 cases where they disagreed (i.e., selected different DMS candidates or None) and resolved them in order to achieve 100% agreement.
This human-in-the-loop methodology helps with the more difficult task of DMS generation, enhancing the overall quality and efficiency the process. The iterative resolution of differences ensured a high quality of annotated data.

§ PREVALENCE OF SE-SPECIFIC FIGURATIVE LANGUAGE

In order to understand if SE-specific figurative language appears frequently in the wild, we examine the frequency of occurrence of figurative language in a large sample of developer communication on GitHub. More specifically, we collected 1,000 issue comments and 1,000 pull request comments for each of the top 100 repositories by star count on GitHub, i.e., a total of 200k comments. We analyzed comments made from September 1, 2022, to January 1, 2023, spanning 4 months, and excluded repositories with fewer than 1,000 issue and pull request comments during this time.
The collected 200k comments were split into a total of 484k sentences using NLTK[<https://www.nltk.org/>]. Leveraging our annotated dataset consisting of 1741 unique figurative expressions (445 SE-specific and 1296 general), we searched for matches in the set of 484k sentences after applying standard NLP pre-processing (removing punctuation and non-alphabet characters, and lemmatization using SpaCy) since some of the figurative expressions have different spelling variations (e.g., root cause, root-cause, and root causes). To ensure that the matches were not spuriously identifying figurative language (due to polysemy), we also executed the metaphor and idiom detection tools <cit.>, the same ones that we use for candidates generation, selecting only the matches that were also confirmed by one of these tools.

Of the examined 484k sentences, the 445 SE-specific figurative expressions that annotators identified occurred in 44k sentences (9%), while the 1296 general figurative expressions occurred in 107k sentences (22%). Some sentences (2.67%) had both SE-specific and general figurative expressions.

The distribution of general and SE-specific figurative expressions in different GitHub repositories is shown in Figure <ref>. SE-specific figurative language does appear in non-trivial amounts in most repositories we examined (i.e., in between 3.69% and 16.08% of sentences), but much less often than general figurative expressions, which occurred in between 13.2% and 38.62% of sentences.
In Figure <ref>, we present the frequency of SE-specific figurative expressions identified within our corpus of 200k GitHub comments. Of the 445 SE-specific figurative expressions we investigated, 324 (72.8%) appear no more than 10 times, as indicated by the red dotted line, suggesting that most such expressions are infrequent. Among these, 193 expressions are absent from our dataset. This absence can be attributed to two main factors: a) expressions that are specific to particular projects (e.g., `ghost highlight' and `ghost monitor' in UI-related projects); and b) unique expressions used to describe highly specific scenarios (e.g., `dead fork,' `magic code').

Percent of sentences with figurative expressions in repos.

Frequency of sentences with SE-specific figurative expressions.

Distribution of figurative language occurrence in GitHub sentences (200k GitHub comments, 484k sentences).

Note that our study provides only a lower bound, as it matches using an incomplete set of figurative language expressions. Therefore, the likely presence of figurative language is even higher than we report. This exploratory study highlights the importance of understanding figurative language in the SE context, as it can provide insight into the daily communications of developers.

§ EXPERIMENTS AND DISCUSSION

Using the assembled dataset, we created specific experiments for each of our three research questions. In this section, we describe the experiments and discuss the corresponding results.

Percent of EMS with a higher similarity to the original sentence than corresponding DMS (Sim$_{EMS}$ > Sim$_{DMS}$).
Model     3c|SE-specific     3c|General     3c|Overall
Sim$_{EMS}$ > Sim$_{DMS}$     p-value     |Cliff's $\delta$|     Sim$_{EMS}$ > Sim$_{DMS}$     p-value     |Cliff's $\delta$|     Sim$_{EMS}$ > Sim$_{DMS}$     p-value     |Cliff's $\delta$|

BERT     84.51%       $p$ < 0.01     0.629     87.40%       $p$ < 0.01     0.638     86.57%      $p$ < 0.01     0.637
RoBERTa     83.70%     $p$ < 0.01     0.648     85.21%     $p$ < 0.01      0.620     84.95%     $p$ < 0.01      0.632
ALBERT     81.79%     $p$ < 0.01     0.610     85.80%     $p$ < 0.01      0.598     85.00%     $p$ < 0.01      0.605
CodeBERT     77.99%     $p$ < 0.01     0.498     79.63%     $p$ < 0.01      0.493     79.11%     $p$ < 0.01      0.495

§.§ RQ1: How well can existing LLMs interpret figurative language (i.e., metaphors and idioms) used in software engineering?

In order to determine how well popular Large Language Models (LLMs) understand metaphors and idioms, we examine whether they can understand the semantic relationship between the original sentence, the equivalent sentence (i.e., EMS), and the different-meaning sentence (i.e., DMS). The task of differentiating EMSs from DMSs of the original sentences can be thought of as Recognizing Textual Entailment (RTE) <cit.>. RTE involves determining whether a statement, called the hypothesis, can be inferred from a given text, called the premise. In our context, the premise is the original sentence, and the hypotheses are the EMS and DMS. We evaluate whether an LLM can infer the EMS from the original sentence, and if it is a DMS, the model should not deduce it.

One way to approximate the RTE task is to posit that the model should recognize the original sentence as semantically closer to the EMS than to the DMS. By comparing the embedding vectors of the original sentence and the sentence in question, we can measure whether the two sentences are similar or dissimilar and, therefore, whether the most similar sentence is an EMS or a DMS.

§.§.§ Compared LLMs
We compare three LLMs – BERT <cit.>, RoBERTa <cit.>, and ALBERT <cit.>, which are popular in NLP and SE tasks <cit.>. BERT <cit.> is a transformer-based model pre-trained on extensive text data from Wikipedia and BooksCorpus.
RoBERTa <cit.> is an improved version of BERT, while ALBERT <cit.> enhances efficiency through parameter reduction techniques.
Additionally, we evaluate a popular SE-specific LLM – CodeBERT <cit.>, which is pre-trained on natural language - programming language pairs. We use bert-base-uncased, roberta-base, albert-base-v2 and microsoft/codebert-base available from Hugging Face[<https://huggingface.co>].

RQ1 evaluation pipeline.

§.§.§ Procedure

We generate embedding vectors using each of the LLMs for each pair of sentences, i.e., <original sentence, EMS> and <original sentence, DMS>. Next, we compute the similarity between the vectors in each pair and then compare the resulting similarity scores <cit.>. We perform standard software engineering text-specific preprocessing operations such as URL removal, username removal, stack trace removal, etc. <cit.>

Since the LLMs in our cohort produce word-level embedding vectors, there are several possibilities for aggregating these into sentence-level embedding vectors. Reimers et al. <cit.> noted that mean pooling (the mean of all per-word output vectors generated by the LLM) is one of the best strategies. While we opt for mean pooling when generating each sentence's embedding vector, we also note that this strategy is still error-prone due to the anisotropy problem, i.e., the difference in the scale of the embedding vectors <cit.>.
For this reason, we apply the normalization proposed by Yan et al. <cit.>, which is based on Singular Value Transformation (SVT). SVT  uses singular value decomposition and a threshold using the soft-exponential function by Godfrey et al. <cit.>

Following normalization, we compute vector pair similarity with the cosine similarity metric.
Cosine similarity measures vector alignment, with values from 1 (identical) to 0 (orthogonal) to -1 (opposite) <cit.>.
Then, we compare the two similarities in order to determine if the <original sentence, EMS> similarity (Sim$_{EMS}$) is higher than the <original sentence, DMS> similarity (Sim$_{DMS}$). Figure <ref> summarises the entire procedure.
To evaluate the RQ, we compute the percentage of instances where Sim$_{EMS}$ is greater than Sim$_{DMS}$. We examine three sentence categories: a) those containing SE-specific figurative language only (n=371); b) those containing general figurative language only (n=1179); and c) overall, containing either (a) and (b), or both (n=1661).
For each model and category, we measure the statistical significance of the difference between the two cosine similarities using the one-tailed Wilcoxon signed-rank test (i.e., testing if Sim$_{EMS}$ > Sim$_{DMS}$ with statistical significance). We apply the Benjamini-Hochberg correction to control the false discovery rate. A small p-value (e.g., p-value < 0.05) indicates that the difference is unlikely to be due to chance and that there is a statistically significant difference between the two samples. We also compute the effect size, which measures the magnitude of the difference between the two samples, using Cliff's Delta ($\delta$) <cit.>, where |$\delta$| > 0.147, 0.33, and 0.474 indicate small, medium, and large effects respectively.

§.§.§ Results and Discussion
Table <ref> shows the SE-specific, General, and Overall (i.e., combined) results. The higher the percentage of <original sentence, EMS> pairs with larger cosine similarity, i.e., Sim$_{EMS}$ > Sim$_{DMS}$, the better the model is at recognizing figurative language. The results table shows that the BERT and RoBERTa models have the highest percentage of correctly understood pairs for all categories. BERT, RoBERTa, and ALBERT models correctly recognize 84.51%, 83.70%, and 81.79% of sentences containing SE-specific figurative expressions, 87.40%, 85.21%, and 85.80% of General figurative expressions, and 86.57%, 84.95%, and 85.0% of the Overall figurative expressions, respectively.
In the case of CodeBERT, which exhibits the poorest results out of all models, there is no significant difference between SE-specific and General results (77.99% and 79.63% respectively). This is likely because the model is pre-trained with programming-specific data enabling it recognize some software engineering figurative language terms. However, it also likely loses the ability to capture General figurative language, which is present in the other LLMs.
From this study, we observe that all of the models can understand figurative language to a reasonable degree (i.e., ranging between 77.99% to 87.40%).

This is also evident from p-value and Cliff's |$\delta$|. In each case, the statistically significant is with a p-value < 0.01 and a |$\delta$| greater than 0.474 is considered a large effect for all models.
The p-value less than 0.01 indicates that the observed difference in similarity between the two groups is highly unlikely to be due to chance and we conclude that there is a statistically significant difference in similarity between the two sets of sentence pairs.
The large |$\delta$| indicates that the similarity between the two groups (i.e., the sentence pairs in group Sim$_{EMS}$ compared to those in group Sim$_{DMS}$ is substantial. The cosine similarity values in Group Sim$_{EMS}$ are consistently higher than those in Group Sim$_{DMS}$, showing that the sentences in Group Sim$_{EMS}$ are more similar to each other compared to those in Group Sim$_{DMS}$.
Together, these results suggest that the two groups of sentence pairs exhibit a notable and meaningful difference in their similarity scores, and this difference is not likely to be due to random chance.

However, the models still fail to recognize between 18.21% and 12.60% of figurative language instances. It is highly likely that if we can improve the models' understanding of figurative language in such cases, they will function better in their use cases.

§.§ RQ2: Can the performance of software engineering-specific affective analysis be improved by a better insight into figurative language?

Affect analysis involves identifying and evaluating human emotions, feelings, and sentiments expressed through written communication.
Kovecses et al. noted that figurative expressions are vital in expressing emotions  <cit.>, while Mohammad et al. <cit.> observed that metaphorical words tend to contain significantly more emotions than the literal sense of the same words.
In software engineering, affect is often related to the software and its development process, including the emotional states of software developers, productivity, and burnout <cit.>. Thus, identifying and understanding affect is crucial for improving software quality and developer productivity. However, several studies have shown challenges in building reliable tools and datasets for mining emotions and opinions in the SE domain <cit.>. A recent study by Imran et al. found that using figurative language in SE-related text can hinder the accurate identification of emotions  <cit.>, partly motivating our RQ2 investigation.

The use of LLMs has become a widely adopted method for identifying and classifying affective expressions in written text <cit.>. LLMs are usually fine-tuned to address specific affect analysis tasks, such as recognizing sentiment or emotions. Recently, one of the most effective ways to fine-tune an LLM is by applying a contrastive learning approach. This approach uses sets of similar and dissimilar instances to train the model to understand the similar instances and differentiate them from the dissimilar ones <cit.>. To answer this RQ, we leverage contrastive learning as the means for LLMs to better capture the meaning and nuances in figurative language present in GitHub comments.

§.§.§ Compared models
Similar to RQ1, we assess the ability of the same four LLMs — BERT <cit.>, RoBERTa <cit.>, ALBERT <cit.>, CodeBERT <cit.> — with the same model versions as RQ1 from Hugging Face. Previous research shows that BERT, RoBERTa and ALBERT work well in SE affect analysis <cit.>.

§.§.§ Contrastive learning
Contrastive learning is a recently proposed machine learning technique that involves training a model
to distinguish between two or more distinct data points by contrasting their differences <cit.>.
The steps for applying this approach to fine-tune LLMs for understanding figurative language elements in the text can be outlined as follows:

* The LLM is presented with a triplet of anchor, positive and negative samples, which are representative of the figurative language elements to be learned.

* The LLM processes the samples and generates output embeddings for the data triplet.

* A loss function encourages the anchor and positive samples to be closer together and the anchor and negative samples to be further apart in the embedding space.

* The process is repeated until the LLM has learned a satisfactory representation.

To apply contrastive learning, we use the original sentences and EMSs as anchor and positive classes and DMSs as negative classes. In other words, we created <original sentence, EMS, DMS> and <EMS, original sentence, DMS> as a pair of triplets, where the first element in each pair is anchor, the second element is positive, and the third element is negative. There is a total of 3322 such triplets of sentences in our dataset.

We use InfoNCE Loss as our loss function <cit.>. Given the embeddings of an anchor, a positive, and a negative sample denoted as $a$, $p$, and $n$ respectively, the InfoNCE loss is computed as follows:
\text{InfoNCE Loss}(a, p, n) = -\log\left(\frac{e^{\text{sim}(a, p)}}{e^{\text{sim}(a, p)} + e^{\text{sim}(a, n)}}\right)
where $\text{sim}(a, p)$ represents the cosine similarity between the embeddings of the anchor and positive samples, and $\text{sim}(a, n)$ represents the cosine similarity between the embeddings of the anchor and negative samples.
The InfoNCE loss maximizes the log-likelihood of anchor-positive similarity and minimizes anchor-negative similarity. We use the Adam optimizer.

Using contrastive learning, the LLM learns to create embeddings that capture the semantic similarity between the original and EMS while recognizing the semantic differences between the original and DMS.
This allows the LLM to learn a representation that separates the positive and negative samples as much as possible. In this case, the LLM learns to recognize the figurative language elements.

After fine-tuning the models with contrastive learning, we assess their performance in two tasks: emotion recognition and incivility detection.
We compare the performance of these fine-tuned models against baseline models that are not fine-tuned with figurative language.

§.§.§ Datasets
We apply the LLMs to two SE-affect datasets.

Emotion Dataset. Imran et al. <cit.> curated a multi-label emotion dataset that is crawled from GitHub. The dataset consists of 2000 data points and six emotion classes: Anger, Love, Fear, Joy, Sadness, and Surprise. The dataset contains 340 (17.0%) Anger comments, 220 (11.0%) Love comments, 198 (9.9%) Fear comments, 422 (21.1%) Joy comments, 274 (13.7%) Sadness comments, and 328 (16.4%) Surprise comments. The rest of the comments are neutral.

Incivility Dataset. Ferreira et al. <cit.> curated a dataset from GitHub's heated issues for incivility detection. The dataset has three parts: comment level, issue level, and sentence level. We consider only the comment-level dataset in our study, which has three classes: Civil, Uncivil, and Technical. We consider only Civil and Uncivil comments as we are only interested in affective analysis for this RQ. The filtered dataset contains 718 comments, of which 232 (32.3%) are Civil comments, and 486 (67.7%) Uncivil comments.

§.§.§ Procedure and Metrics.

Using random stratified sampling for each class, we divide all two datasets into train (80%) and test (20%) sets <cit.>. For each task (i.e., incivility detection and emotion detection), we train (or fine-tune) both the LLMs' contrastive learning and baseline versions. In other words, the contrastive learning models are fine-tuned twice, first with contrastive learning and figurative language and second with a task-specific dataset. The baselines are only fine-tuned with the task-specific dataset.

We choose a metric that is frequently used to evaluate classification tasks: F1-score, which aggregates Precision and Recall. Precision is the ratio of true positive instances to the total predicted positive instances, and Recall is the ratio of true positive instances to all instances in the positive class. F1-score is the harmonic mean of Precision and Recall:
$\text{\em F1-score} = 2 * \frac{Precision * Recall}{Precision + Recall}$. We also calculate the micro average version for averaging the F1-score across the classes following previous research <cit.>.

§.§.§ Results and Discussion
Emotion Classification: Table <ref> shows the results of the emotion classification task on Imran et al.'s multi-label emotion dataset <cit.>, using BERT, RoBERTa (RBTa), ALBERT (ALBT), CodeBERT (CodBT) and their fine-tuned with figurative language counterparts (BERT-FL, RBTa-FL, ALBT-FL, CodBT-FL). The table presents the F1-score for each emotion class, the micro-averaged F1-score, and the improvement in the F1-score achieved by the figurative language versions of the models.
The results show that the use of contrastive learning with figurative language improves performance on the emotion classification task for most emotion types. For the micro-averaged F1-score, across all emotions, the figurative language versions of the models achieve an improvement of 6.60%, 6.66%, 3.63%, and 3.90% for BERT, RoBERTa, ALBERT, and CodeBERT respectively.
This implies that adding figurative language to these models improves their capability to comprehend and interpret the subtleties that developers use in their communication.

In all four models, we see an increase in True Positives and a decrease in both False Negatives and False Positives. For instance, for the BERT model, across 6 emotions, micro-averaged recall increases by 5.58%, and micro-averaged precision increases by 7.59%. This indicates improved precision in predictions after applying contrastive learning.

When considering the average improvement in individual emotions across all models in Table <ref>, we observe that `Joy' has most improvement (7.67%), followed by `Surprise' (5.45%). This correlates with the frequency of occurrence of these emotions in GitHub comments, e.g., Joy is much more commonly found than Fear.
As our figurative language dataset is randomly sampled, it is likely to contain figurative expressions closely related to the emotions that are more commonly observed in GitHub.
This result suggests that curating a larger and more diverse set of comments that include figurative language could lead to a stronger and more balanced performance improvement.

Evaluation of LLMs finetuned with figurative language on the Emotions Dataset (F1-score).

Model     Anger     Love     Fear     Joy     Sad.     Surp.     Mic.Avg.

BERT     0.506     0.712     0.536     0.579     0.636     0.594      0.588

BERT-FL     0.547     0.709     0.562     0.608     0.661     0.632     0.627

+/-     +8.10%     -0.42%	    +4.85%     	+5.01%     +3.93%     +6.40%     +6.60%

RoBERTa     0.525    	0.683     0.500     0.613     0.673     0.592     0.593

RoBERTa-FL     0.551     0.733     0.545     0.667     0.667     0.617     0.632

+/-    +4.95%	    +6.82%     +8.26%     +8.10%	    -0.90%	    +4.05%     +6.66%

ALBERT     0.462     0.658     0.430     0.487     0.628     0.564      0.531

ALBERT-FL     0.443     0.682     0.435     0.540     0.624     0.592     0.550

+/-     -4.11%	   +3.52%	   +1.15%	   +9.81%	   -0.64%	   +4.73%     +3.63%

CodeBERT     0.484     0.711    	0.507     0.558     0.575     0.576     0.561

CodeBERT-FL     0.497     0.723     0.444     0.605     0.645     0.617     0.583

+/-    +2.79%	   +1.70%	   -14.08%	   +7.75%	   +10.92%	   +6.61%     +3.90%

Avg. +/-    +2.93%	   +2.90%    +0.04%	   +7.67%    +3.33%	   +5.45%     +5.20%

Error Analysis of BERT-FL vs. BERT.
To gain deeper insight into figurative language-based models' predictive accuracy relative to baseline models, we perform qualitative analysis. Our focus is solely on BERT and BERT-FL models' predictions. We examine two specific areas: 1) True Positives where BERT-FL is correct while baseline BERT is not, and 2) True Positives where baseline BERT is correct while BERT-FL is not.

Among the positive instances, BERT-FL correctly predicts 39 utterances that the baseline BERT model does not.
Consider the following sentence: Bah. Wasn't supposed to add anything – it was a debugging leftover.... In this case, BERT-FL correctly predicts `Anger', whereas BERT misclassifies it. Here, the word leftover is used metaphorically. Normally, leftover refers to something that remains unused or unconsumed, particularly in the context of food[<https://www.merriam-webster.com/dictionary/leftover>].
However, in the given sentence, the word is used to imply that some code or modifications were unintentionally left behind or overlooked during the debugging process. The BERT-FL model likely captures the context more effectively. Another example, Oh nice!! I've seen that syntax floating around, wanting to try it for a while `raising-hands` - BERT-FL correctly classifies as `Joy' which the BERT baseline model misclassifies. Here, the BERT-FL is likely able to capture that floating around is an idiom[<https://idioms.thefreedictionary.com/floating+around>] and interpret the meaning. In some instances, BERT-FL makes correct predictions by adopting a more conservative classification approach. For instance, BERT classifies the following sentence as `Anger': Please put this below line 5 (together with the other non-app imports) :pray:. However, BERT-FL accurately predicts that it is not `Anger'.

On the other hand, in 27 cases, BERT-FL makes wrong predictions where BERT does not.
Consider this utterance: I have currently no clue, but I'll have a look, this sentence contains the idioms have a clue [<https://idioms.thefreedictionary.com/have+a+clue>] and have a look[<https://www.thefreedictionary.com/have+a+look>]. The author of the comment likely was puzzled about some functions or errors. BERT identifies correctly as `Surprise' but BERT-FL does not. Possibly, BERT-FL interpreted these idiomatic expressions more of a literal interpretation of the words. In some cases, BERT-FL just misclassifies without any involvement of any figurative expressions. For example, I guess my concern is that it sets a precedent where somebody could see it and think that it would be fine to use in `core'. This expression express concern which is annotated as `Fear'. This expression conveys concern, annotated as `Fear'. BERT identifies it correctly, but BERT-FL does not.
It is possible that during the contrastive learning process, BERT-FL may lose some of the baseline BERT model's ability to capture nuanced emotional indicators in certain sentences accurately.
This suggests that while this approach improves the overall model performance but may introduce limitations or biases in some cases.

Evaluation of LLMs finetuned with figurative language on the Incivility Dataset (F1-score).

Model     Civil     Uncivil     Micro Average

BERT     0.537     0.814     0.734

BERT-FL     0.587     0.853     0.783

+/-     +8.54%	    +4.84%     +6.67%

RoBERTa     0.424     0.827     0.734

RoBERTa-FL     0.535     0.847     0.769

+/-     +20.73%	   +2.33%    +4.76%

ALBERT     0.151     0.807     0.685

ALBERT-FL     0.423     0.809     0.713

+/-     +64.28%	    +0.30%     +4.08%

CodeBERT     0.185     0.810     0.692

CodeBERT-FL     0.431     0.833     0.741

+/-     +57.01%	   +2.74%     +7.07%

Avg. +/-     +37.64%     +2.55%     +5.65%

Incivility Classification: Table <ref> presents the results of the incivility classification task on Ferreira et al.'s incivility dataset <cit.>, using the same four large language models (BERT, RoBERTa, ALBERT, and CodeBERT) with and without the contrastive learning approach.
The micro-averaged F1-scores indicate that the models perform better when applying the contrastive learning approach. Overall, the BERT, RoBERTa, ALBERT, and CodeBERT models have an average improvement of 6.67%, 4.76%, 4.08%, and 7.07% respectively, when the contrastive learning approach is applied.
Since the incivility dataset is small and imbalanced, the baseline models often struggle to classify the minor `Civil' class, except for BERT.

We also observed a significant average improvement of 37.64% across all models in the `Civil' class, compared to a modest 2.55% improvement in the `Uncivil' class. This discrepancy likely arises because the figurative language dataset used for contrastive learning primarily consists of `Civil' comments, which are much more common on GitHub than `Uncivil' comments. Incorporating more figurative expressions from `Uncivil' comments into the dataset could potentially enhance performance in this category as well.

It is important to note that the substantial improvements in identifying `Civil' comments are largely attributable to ALBERT and CodeBERT, which showed improvements of 180% and 133%, respectively. These models started from a lower performance baseline, making such large gains more achievable compared to other models. However, BERT and RoBERTa also demonstrated stronger performance improvements in the `Civil' class.

§.§ RQ3: Can a better understanding of figurative language enhance software engineering automation where affect plays a role?

To answer this RQ, we focus on a specific use case: automatic bug report priority detection, a major research area in software engineering <cit.>, where previous research has highlighted the role of affect <cit.>.

Dataset. Bugzilla bug reports are widely used for priority detection <cit.>. The bug priority reports in Bugzilla are divided into 5 classes (i.e., P1 to P5, where P1 represents the highest priority while P5 represents the lowest priority).
Wang et al. collected 220k bug reports from Bugzilla <cit.>. We sample 25% of this dataset using stratified sampling across the 5 classes. We sample separately from the training and testing splits provided by the authors, which yielded a total of 49.6k bug reports. The distributions provided by the authors are: 1) training: P1 - 19.56%, P2 - 18.45%, P3 - 58.12%, P4 - 1.66%, and P5 - 2.21%; and 2) testing: P1 - 19.21%, P2 - 17.66%, P3 - 59.5%, P4 - 1.48%, and P5 - 2.15%.

Procedure and Metrics. We use the same four LLMs (BERT, RoBERTa, ALBERT, and CodeBERT) as baselines and follow the same approach for training and testing described in RQ2. We use F1-score as evaluation metric.

Results and Discussion. Table <ref> shows the results of bug report priority prediction on the Bugzilla dataset. All four models made small improvements (1.96%, 2.40%, 3.71%, and 1.61% respectively) when fine-tuned with figurative languages.
On the other hand, the improvements across classes (P1-P5) varied. The change in the P5 class was minimal (0.27%), and none of the models succeeded in recognizing any of the P4 instances. This is likely due to the fact that these two classes have the smallest amounts of data, comprising only 1.66% for P4 and 2.21% for P5 of the training data, respectively. Such findings suggest that fine-tuning with figurative language is not beneficial in cases of extreme data imbalance. For the average performance improvement across all models in the other three bug priority classes, we observe that P3 improved least (1.43%) while P1 and P2 make more substantial gains of 4.37% and 8.23%. Umer et al. <cit.> noted that a substantial number of instances in the Bugzilla dataset are `Neutral', indicating that including figurative expressions from `Neutral' utterances — which our dataset predominantly omits — could potentially yield additional benefits.

Evaluation  of LLMs finetuned with figurative language on the Bug Report Priority dataset (F1-score).

Model     P1     P2     P3     P4     P5      Micro Average

BERT     0.606     0.329     0.833     0.0     0.663     0.716

BERT-FL     0.632     0.359     0.842     0.0     0.667     0.730

+/-     +4.31%     +9.14%     +1.10%     -     +0.52%     +1.96%

RoBERTa     0.61     0.293     0.827     0.0     0.677     0.707

RoBERTa-FL     0.624     0.343     0.839     0.0     0.674     0.724

+/-     +1.91%	   17.24%	   +1.39%	   -    -0.51%     +2.40%

ALBERT     0.564     0.288     0.810     0.0     0.670     0.683

ALBERT-FL     0.602     0.299     0.827     0.0     0.674     0.709

+/-     +6.71%    +3.88%	   +2.14%     -    +0.53%    +3.71%

CodeBERT     0.608     0.363     0.830     0.0     0.667      0.714

CodeBERT-FL     0.636	    0.373      0.839     0.0      0.670      0.726

+/-     +4.55%	   2.64%	   +1.08%    -	   0.52%    +1.61%

Avg. +/-     +4.37%	    +8.23%	    +1.43%     -     +0.27%      +2.42%

Error Analysis of BERT-FL vs. BERT. To get an understanding of where fine-tuned models are getting results correctly compared to baseline models, we look into 51 instances where BERT-FL makes the right predictions but BERT does not. We find that, indeed, some of these bug reports include metaphors and idioms.
For example, consider the following bug report description, which is at the P2 priority level: Deadlock when adding JSF framework I have experienced a deadlock while I was adding JSF framework to regular web project. [...] Here Deadlock is a SE-specific figurative expression. The baseline model predicted P3, but BERT-FL made the correct prediction.
Another example Toot your own horn, put your name in the credits window The credits window is empty [...], annotated as P3. Here, toot your own horn is an idiom. BERT-FL correctly predicted but the baseline model did not.

However, there are also cases with figurative language where the fine-tuned model predicted incorrectly, while the baseline model was right.
For example, consider the following bug report offline task data is not retrieved on query [...] (i.e., fetch all things before hitting the road). [...] Here, hitting the road is an idiom[<https://www.thefreedictionary.com/hitting+the+road>].
The BERT-FL model predicts P1 when the original label is P3. It is possible that BERT-FL recognizes the idiom, prioritizes its figurative meaning, and predicts a higher class than the original label.

§.§ Implications

There are a number of actionable implications to our study. Creating a glossary of common figurative language for a software project can be an invaluable tool for efficiently onboarding new developers <cit.>. It would help newcomers understand project-specific or domain-specific terms, which are essential for their quick integration. Minimizing the use of obscure jargon that may cause misunderstandings can enhance mutual understanding and collaboration among project participants <cit.>. Lastly, it's important to consider cultural differences <cit.> that may influence the interpretation of figurative language, as these nuances can significantly affect comprehension and communication within a diverse team.

Our study paves the way for several promising research directions in the realm of figurative language comprehension within software engineering: 1) Integrating figurative language into cutting-edge software engineering tools, such as CleBPI <cit.>, could be achieved through innovative approaches like contrastive learning, self-supervised learning, or adversarial training; 2) Investigating the role of figurative language in specific scenarios, including toxic or uncivil comments, bug reports, and documentation, may yield insights into its effects on software development workflows; 3) Exploring the use of figurative language as a means for data augmentation presents an intriguing opportunity, building on established data augmentation techniques <cit.>; 4) Broadening the scope of analysis to encompass various forms of figurative language, such as similes, hyperbole, and personification, could enhance the depth of model training; 5) Extending our analysis to software engineering communication platforms beyond GitHub, including Stack Overflow, Gitter, JIRA, and app reviews, would offer a more holistic view of figurative language usage across different settings. Adapting Large Language Models (LLMs) for domain-specific figurative language has recently garnered interest in the NLP community <cit.>. Our work compliments this by adapting LLMs to the figurative language in software engineering.

§ RELATED WORK

We describe the related work sourced from three different domains: figurative language analysis in the domain of Natural Language Processing (NLP), affect analysis in Software Engineering (SE), and bug report analysis in SE.

Figurative Language in NLP. Figurative language has long been a topic of study in the field of NLP <cit.>. Research has explored its impact across various communication channels, including online reviews and social media <cit.>. Social media platforms frequently employ figurative language to convey emotions, opinions, and feelings <cit.>.
Furthermore, there have been investigations that focus on the use of figurative language within specific domains <cit.>.
Specific forms of figurative language, such as metaphors, idioms, similes, sarcasm, and irony, have been studied in relation to tasks like offensive language detection <cit.>.

Scholars have categorized the detection of figurative language into two tasks: recognizing text containing figurative language and interpreting figurative expressions to identify their intended literal meaning <cit.>. Recognizing figurative language presents challenges due to the multiple interpretations that expressions can have. To address these challenges, various methods have been proposed, such as word vectors, rule-based approaches, semantic patterns, and the application of LLMs <cit.>. Interpreting figurative language is a more complex task that requires a deeper understanding of the text's meaning <cit.>. Previous approaches have utilized knowledge-based and corpus-based methods <cit.>. Researchers have leveraged LLMs to paraphrase figurative expressions tasks and have been successful in interpreting metaphors, idioms, hyperbole, irony, sarcasm, and similes <cit.>. Some of the most common strategies applied to interpret figurative expressions using LLMs are zero-shot learning and fine-tuning using contrastive learning <cit.>. Different from the prior work, we investigate the ability of LLMs to interpret figurative language in the context of SE communication.

Affect Analysis in SE. Affective expressions in written text can be effectively analyzed by identifying linguistic cues that convey emotions, feelings, or attitudes <cit.>. The field of affect analysis in software engineering is rapidly growing, focusing on understanding how emotions, opinions, sentiment, toxicity, incivility, burnout, and offensive language impact software development activities <cit.>.

LLMs have emerged as powerful tools for affect analysis, making significant strides in the software engineering domain <cit.>. They have proven their mettle in sentiment analysis across various software-related artifacts and have even been instrumental in detecting incivility and toxicity <cit.>.

Despite the advancements made and the use of modern LLMs, some limitations need to be addressed, particularly in terms of generalizability.
SE-specific affect analysis tools trained on one communication forum may not perform well when applied to another forum due to differences in norms, conventions, and cultures that influence the expression of emotions and sentiments <cit.>. One major reason for this limitation is the tools' inability to recognize implicit emotions or sentiments, often inferred through context, tone, or other cues <cit.>. These challenges call for developing more versatile and adaptable tools that can be applied across multiple domains. In this paper, we explore interpreting and fine-tuning figurative languages with LLMs to enhance the generalizability of SE-specific affect analysis tools across different artifacts.

Bug Report Analysis in SE. Bug report analysis is a mature research area in SE spanning tasks like duplicate bug detection, bug localization, deficient bug report, bug severity prediction, and priority assignment <cit.>. Of particular relevance is bug report priority prediction, where affects in report descriptions can influence triage decisions <cit.>. Recently, priority inference models based on deep learning have been proposed using LLMs <cit.>. This study explores whether fine-tuning LLMs with figurative language can enhance performance of the task or not.

§ THREATS TO VALIDITY
Several limitations may impact the interpretation of our findings. We categorize and list each of them below.

Construct validity. Construct validity refers to the degree to which the study measures the concepts and constructs it claims to measure.
A threat may arise from the manual annotations for the dataset, specifically in creating semantically similar EMS and DMS sentences. To mitigate this, we provided clear instructions and examples to the annotators.
Additionally, we only examined metaphors and idioms; including other figurative language may alter results. To investigate this, our annotation approach can be expanded to analyze other forms.
Another potential threat is that our figurative language dataset was sourced from developer communication in 9 GitHub repositories, which may not be representative of the figurative language present on GitHub.

Internal validity. Internal validity concerns the extent to which the study's findings can be attributed to the manipulation of the independent variable.
A threat is that the improved affect analysis performance with figurative language fine-tuning may not be solely due to the figurative language. However, we see consistent improvements across all models and datasets, indicating it is a key factor.
Not doing cross-validation on the smaller datasets can be another threat. To mitigate this, we use stratified sampling for representativeness and a standard 80-20% train-test split.

External validity. External validity pertains to the generalization of the findings of our study to other settings and contexts.
Our results may not generalize beyond the specific studied models, datasets, and any other domain than GitHub. However, we use diverse pre-trained LLMs and a Bugzilla dataset, showing some cross-domain applicability. Further investigation is needed to validate our results beyond the tools, data, and platforms used in our study.

§ CONCLUSION

This paper examined the relevance and impact of figurative language in software engineering communication. To investigate this, we annotated metaphors and idioms in a set of 2000 sentences collected from GitHub issues and PRs which resulted in 1661 sentences with figurative expressions, conducted a comprehensive analysis of the prevalence of figurative language in messages posted on PRs and issues in top 100 GitHub repositories, fine-tuned several state-of-the-art pre-trained LLMs with the annotated dataset, and evaluated the performance of these fine-tuned models on three publicly available SE-specific datasets. Our results indicated that figurative language is prevalent in software engineering communication, and fine-tuning LLMs with figurative language leads to improved performance on affect analysis tasks (on the best model, 6.66% improvement on a GitHub emotion dataset, 7.07% improvement on a GitHub incivility dataset, and 3.71% improvement on a bug report prioritization dataset). Overall, our findings provide evidence for the relevance and impact of figurative language in software engineering communication and the potential benefits of fine-tuning LLMs with figurative language in the context of software engineering. However, there is room for further investigation.

Beyond the future work directions discussed in Section <ref>, our error analysis shows that fine-tuned models may sometimes overemphasize figurative language, motivating the need for a different fine-tuning approach. Addressing this issue while preserving interpretive abilities presents an area for future research. Experimenting with generative language models like ChatGPT and LLaMa to assess their potential in enhancing the automatic interpretation of complex figurative expressions could significantly benefit communication and understanding in software development contexts.
Overall, this study provides a starting point for further empirical research on figurative language’s impact on software engineering communications in different application domains.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Acheampong et al(2021)]
authorpersonFrancisca Adoma Acheampong,
personHenry Nunoo-Mensah, and personWenyu Chen.
Transformer Models for Text-Based Emotion
Detection: A Review of BERT-Based Approaches.
journalArtif. Intell. Rev. volume54,
number8 (datedec year2021),

[Alexandru et al(2018)]
authorpersonCarol V. Alexandru,
personJosé J. Merchante, personSebastiano
Panichella, personSebastian Proksch, personHarald C.
Gall, and personGregorio Robles.
On the Usage of Pythonic Idioms. In
booktitleProceedings of the 2018 ACM SIGPLAN International
Symposium on New Ideas, New Paradigms, and Reflections on Programming and
Software. publisherACM, addressUSA,

[Batra et al(2021)]
authorpersonHimanshu Batra,
personNarinder Singh Punn, personSanjay Kumar
Sonbhadra, and personSonali Agarwal.
BERT-Based Sentiment Analysis: A Software
Engineering Perspective. In booktitleDatabase and Expert
Systems Applications: 32nd International Conference, DEXA 2021.
publisherSpringer-Verlag, addressBerlin, Heidelberg,

[Biswas et al(2020)]
authorpersonE. Biswas, personM. Karabulut,
personL. Pollock, and personK. Vijay-Shanker.
Achieving Reliable Sentiment Analysis in the
Software Engineering Domain using BERT. In booktitle2020
IEEE ICSME. publisherIEEE Computer Society,
addressUSA, pages162–173.

[Botev and Ridder(2017)]
authorpersonZdravko Botev and personAd
Ridder. year2017.
Variance reduction.
journalWiley statsRef: Statistics reference online
(year2017), pages1–6.

[Bowman et al(2015)]
authorpersonSamuel R. Bowman, personGabor
Angeli, personChristopher Potts, and
personChristopher D. Manning. year2015.
A large annotated corpus for learning natural
language inference. In booktitleProceedings of the 2015
Conference on EMNLP. publisherACL, addressLisbon,
Portugal, pages632–642.

[Bowman et al(2020)]
authorpersonSamuel R. Bowman,
personJennimaria Palomaki, personLivio
Baldini Soares, and personEmily Pitler.
New Protocols and Negative Results for Textual
Entailment Data Collection. In booktitleProceedings of the
2020 Conference on EMNLP. publisherACL,
addressOnline, pages8203–8214.

[Briskilal and Subalalitha(2022)]
authorpersonJ Briskilal and personCN
Subalalitha. year2022.
An ensemble model for classifying idioms and
literal texts using BERT and RoBERTa.
journalInformation Processing & Management
volume59, number1 (year2022),

[Chakrabarty et al(2021)]
authorpersonTuhin Chakrabarty,
personDebanjan Ghosh, personAdam Poliak, and
personSmaranda Muresan. year2021.
Figurative Language in Recognizing Textual
Entailment. In booktitleFindings of the ACL: ACL-IJCNLP
2021. publisherACL, addressOnline,

[Chakrabarty et al(2022)]
authorpersonTuhin Chakrabarty, personArkadiy
Saakyan, personDebanjan Ghosh, and personSmaranda
Muresan. year2022.
FLUTE: Figurative Language Understanding through
Textual Explanations. In booktitleProceedings of the 2022
Conference on EMNLP. publisherACL, addressAbu Dhabi,
United Arab Emirates, pages7139–7159.

[Chaparro et al(2019)]
authorpersonO. Chaparro, personJ. Florez,
personU. Singh, and personA. Marcus.
Reformulating Queries for Duplicate Bug Report
Detection. In booktitle2019 IEEE 26th International
Conference on Software Analysis, Evolution and Reengineering (SANER).
publisherIEEE Computer Society, addressUSA,

[Chaparro et al(2017)]
authorpersonOscar Chaparro, personJing Lu,
personFiorella Zampetti, personLaura Moreno,
personMassimiliano Di Penta, personAndrian Marcus,
personGabriele Bavota, and personVincent Ng.
Detecting Missing Information in Bug Descriptions.
In booktitleProceedings of the 2017 11th Joint Meeting on
Foundations of Software Engineering. publisherACM,
addressUSA, pages396–407.

[Chatterjee et al(2021b)]
authorpersonPreetha Chatterjee,
personKostadin Damevski, personNicholas A. Kraft,
and personLori Pollock. year2021b.
Automatically Identifying the Quality of Developer
Chats for Post Hoc Use.
journalACM TOSEM volume30,
number4, Article articleno48 (datejul
year2021), numpages28 pages.

[Chatterjee et al(2021a)]
authorpersonPreetha Chatterjee,
personKostadin Damevski, and personLori L.
Pollock. year2021a.
Automatic Extraction of Opinion-based Q&A from
Online Developer Chats. In booktitle43rd IEEE/ACM
International Conference on Software Engineering, ICSE 2021.
publisherIEEE, addressMadrid, Spain,

[Chen et al(2017)]
authorpersonC. Chen, personZ. Xing, and
personX. Wang. year2017.
Unsupervised Software-Specific Morphological Forms
Inference from Informal Discussions. In booktitle2017
IEEE/ACM 39th ICSE. publisherIEEE Computer Society,
addressUSA, pages450–461.

[Chen et al(2019b)]
authorpersonXiang Chen, personChunyang
Chen, personDun Zhang, and personZhenchang Xing.
Sethesaurus: Wordnet in software engineering.
journalIEEE Transactions on Software Engineering
volume47, number9 (year2019),

[Chen et al(2019a)]
authorpersonZhenpeng Chen, personYanbin
Cao, personXuan Lu, personQiaozhu Mei, and
personXuanzhe Liu. year2019a.
SEntiMoji: An Emoji-Powered Learning Approach for
Sentiment Analysis in Software Engineering. In
booktitleProceedings of the 2019 27th ACM Joint Meeting on
European Software Engineering Conference and Symposium on the Foundations of
Software Engineering. publisherACM, addressUSA,

[Chen et al(2021)]
authorpersonZhenpeng Chen, personYanbin
Cao, personHuihan Yao, personXuan Lu,
personXin Peng, personHong Mei, and
personXuanzhe Liu. year2021.
Emoji-powered sentiment and emotion detection from
software developers’ communication data.
journalACM TOSEM volume30,
number2 (year2021), pages1–48.

[Chiorrini et al(2021)]
authorpersonAndrea Chiorrini, personClaudia
Diamantini, personAlex Mircoli, and personDomenico
Potena. year2021.
Emotion and sentiment analysis of tweets using
BERT. In booktitleProceedings of the Workshops of the
EDBT/ICDT 2021 Joint Conference (seriesCEUR Workshop
Proceedings, Vol. volume2841).
publisherCEUR-WS.org, addressNicosia, Cyprus.

[Ciborowska and Damevski(2022)]
authorpersonAgnieszka Ciborowska and
personKostadin Damevski. year2022.
Fast Changeset-Based Bug Localization with BERT.
In booktitleProceedings of the 44th ICSE.
publisherACM, addressUSA, pages946–957.

[Ciniselli et al(2021)]
authorpersonMatteo Ciniselli, personNathan
Cooper, personLuca Pascarella, personDenys
Poshyvanyk, personMassimiliano Di Penta, and
personGabriele Bavota. year2021.
An Empirical Study on the Usage of BERT Models
for Code Completion. In booktitle18th IEEE/ACM
International Conference on MSR. publisherIEEE,
addressMadrid, Spain, pages108–119.

authorpersonNorman Cliff.
Dominance statistics: Ordinal analyses to answer
ordinal questions.
journalPsychological Bulletin
volume114 (year1993), pages494–509.

[del Arco et al(2022)]
authorpersonFlor Miriam Plaza del Arco,
personM. Dolores Molina-González, personL. Alfonso
Ureña-López, and personMaría-Teresa Martín-Valdivia.
Integrating implicit and explicit linguistic
phenomena via multi-task learning for offensive language detection.
journalKnowledge-Based Systems
volume258 (year2022), pages109965.

[Devlin et al(2019)]
authorpersonJacob Devlin, personMing-Wei
Chang, personKenton Lee, and personKristina
Toutanova. year2019.
BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. In
booktitleProceedings of the 2019 Conference of the NAACL:
Human Language Technologies. publisherACL,
addressMinneapolis, Minnesota, pages4171–4186.

[Do Dinh et al(2018)]
authorpersonErik-Lân Do Dinh,
personHannah Wieland, and personIryna Gurevych.
Weeding out Conventionalized Metaphors: A Corpus of
Novel Metaphor Annotations. In booktitleProceedings of the
2018 Conference on EMNLP. publisherACL,
addressBrussels, Belgium, pages1412–1424.

[Dominic et al(2020)]
authorpersonJ. Dominic, personC. Ritter,
and personP. Rodeghero. year2020.
Onboarding Bot for Newcomers to Software
Engineering. In booktitle2020 IEEE/ACM International
Conference on Software and System Processes (ICSSP).
publisherIEEE Computer Society, addressUSA,

[Esmaeilzadeh and Taghva(2022)]
authorpersonArmin Esmaeilzadeh and
personKazem Taghva. year2022.
Text Classification Using Neural Network Language
Model (NNLM) and BERT: An Empirical Comparison. In
booktitleIntelligent Systems and Applications,
editorpersonKohei Arai (Ed.).
publisherSpringer International Publishing,
addressCham, pages175–189.

authorpersonKawin Ethayarajh.
How Contextual are Contextualized Word
Representations? Comparing the Geometry of BERT, ELMo, and GPT-2
Embeddings. In booktitleProceedings of the 2019 Conference
on EMNLP-IJCNLP. publisherACL, addressHong Kong,
China, pages55–65.

[Feng et al(2020)]
authorpersonZhangyin Feng, personDaya Guo,
personDuyu Tang, personNan Duan,
personXiaocheng Feng, personMing Gong,
personLinjun Shou, personBing Qin,
personTing Liu, personDaxin Jiang, and
personMing Zhou. year2020.
CodeBERT: A Pre-Trained Model for Programming
and Natural Languages. In booktitleFindings of the ACL:
EMNLP 2020. publisherACL, addressOnline,

[Ferreira et al(2022)]
authorpersonI. Ferreira, personB. Adams,
and personJ. Cheng. year2022.
How heated is it? Understanding GitHub locked
issues. In booktitle2022 IEEE/ACM 19th International
Conference on Mining Software Repositories. publisherIEEE
Computer Society, addressUSA, pages309–320.

[Ferreira et al(2021)]
authorpersonIsabella Ferreira, personJinghui
Cheng, and personBram Adams. year2021.
The "Shut the F**k up" Phenomenon: Characterizing
Incivility in Open Source Code Review Discussions.
journalProc. ACM Hum.-Comput. Interact.
volume5, numberCSCW2, Article
articleno353 (dateoct year2021),
numpages35 pages.

[Ferreira et al(2023)]
authorpersonIsabella Ferreira, personAhlaam
Rafiq, and personJinghui Cheng.
Incivility detection in open source code review and
issue discussions.
journalJournal of Systems and Software
(year2023), pages111935.

[Fussell and Moss(2014)]
authorpersonSusan R Fussell and
personMallie M Moss. year2014.
Figurative language in emotional communication.
In booktitleSocial and cognitive approaches to
interpersonal communication. publisherPsychology Press,

[Gamage et al(2022)]
authorpersonGihan Gamage, personDaswin De
Silva, personAchini Adikari, and personDamminda
Alahakoon. year2022.
A BERT-based Idiom Detection Model. In
booktitle15th International Conference on Human System
Interaction. publisherIEEE, addressMelbourne,
Australia, pages1–5.

[Gilardi et al(2023)]
authorpersonFabrizio Gilardi, personMeysam
Alizadeh, and personMaël Kubli.
ChatGPT outperforms crowd workers for
text-annotation tasks.
journalProceedings of the National Academy of
Sciences volume120, number30
(year2023), pagese2305016120.

[Giora and Fein(1999)]
authorpersonRachel Giora and personOfer
Fein. year1999.
On understanding familiar and less-familiar
figurative language.
journalJournal of Pragmatics volume31,
number12 (year1999), pages1601–1618.

[Godfrey and Gashler(2015)]
authorpersonL. B. Godfrey and personM. S.
Gashler. year2015.
A continuum among logarithmic, linear, and
exponential functions, and its potential to improve generalization in neural
networks. In booktitle2015 7th International Joint
Conference on Knowledge Discovery, Knowledge Engineering and Knowledge
Management (IC3K). publisherIEEE Computer Society,
addressUSA, pages481–486.

[Gomaa et al(2013)]
authorpersonWael H Gomaa, personAly A
Fahmy, et al year2013.
A survey of text similarity approaches.
journalinternational journal of Computer
Applications volume68, number13
(year2013), pages13–18.

[Gomes et al(2019)]
authorpersonLuiz Alberto Ferreira Gomes,
personRicardo da Silva Torres, and
personMario Lúcio Côrtes. year2019.
Bug report severity level prediction in open source
software: A survey and research opportunities.
journalInformation and Software Technology
volume115 (year2019), pages58–78.

[Graziotin et al(2017)]
authorpersonDaniel Graziotin, personFabian
Fagerholm, personXiaofeng Wang, and personPekka
Abrahamsson. year2017.
On the Unhappiness of Software Developers. In
booktitleProceedings of the 21st International Conference on
Evaluation and Assessment in Software Engineering.
publisherACM, addressUSA, pages324–333.

[Graziotin et al(2015)]
authorpersonDaniel Graziotin, personXiaofeng
Wang, and personPekka Abrahamsson.
Do Feelings Matter? On the Correlation of Affects
and the Self-Assessed Productivity in Software Engineering.
journalJ. Softw. Evol. Process
volume27, number7 (datejul
year2015), pages467–487.

authorpersonPragglejaz Group.
MIP: A Method for Identifying Metaphorically Used
Words in Discourse.
journalMetaphor and Symbol volume22,
number1 (year2007), pages1–39.

[Haagsma et al(2020)]
authorpersonHessel Haagsma, personJohan
Bos, and personMalvina Nissim.
MAGPIE: A Large Corpus of Potentially Idiomatic
Expressions. In booktitleProceedings of the Twelfth
Language Resources and Evaluation Conference. publisherEuropean
Language Resources Association, addressMarseille, France,

[Hao and Veale(2010)]
authorpersonYanfen Hao and personTony
Veale. year2010.
An ironic fist in a velvet glove: Creative
mis-representation in the construction of ironic similes.
journalMinds and Machines  volume20
(year2010), pages635–650.

[Hilton et al(2022)]
authorpersonKelsey Hilton, personAkbar
Siami Namin, and personKeith S Jones.
Metaphor identification in cybersecurity texts: a
lightweight linguistic approach.
journalSN Applied Sciences volume4,
number2 (year2022), pages60.

[Huang et al(2023)]
authorpersonFan Huang, personHaewoon Kwak,
and personJisun An. year2023.
Is ChatGPT Better than Human Annotators? Potential
and Limitations of ChatGPT in Explaining Implicit Hate Speech. In
booktitleCompanion Proceedings of the ACM Web Conference
2023. publisherACM, addressUSA,

[Imran et al(2021)]
authorpersonMia Mohammad Imran,
personAgnieszka Ciborowska, and personKostadin
Damevski. year2021.
Automatically Selecting Follow-up Questions for
Deficient Bug Reports. In booktitle18th IEEE/ACM
International Conference on Mining Software Repositories.
publisherIEEE, addressMadrid, Spain,

[Imran et al(2023)]
authorpersonMia Mohammad Imran,
personYashasvi Jain, personPreetha Chatterjee, and
personKostadin Damevski. year2023.
Data Augmentation for Improving Emotion Recognition
in Software Engineering Communication. In
booktitleProceedings of the 37th IEEE/ACM International
Conference on Automated Software Engineering. publisherACM,
addressUSA, Article articleno29,
numpages13 pages.

[Islam and Zibran(2017)]
authorpersonMd Rakibul Islam and
personMinhaz F. Zibran. year2017.
Leveraging automated sentiment analysis in software
engineering. In booktitleProceedings of the 14th
International Conference on Mining Software Repositories,
editorpersonJesús M. González-Barahona,
personAbram Hindle, and personLin Tan (Eds.).
publisherIEEE Computer Society, addressBuenos Aires,
Argentina, pages203–214.

[Joseph et al(2023)]
authorpersonRohan Joseph, personTimothy
Liu, personAik Beng Ng, personSimon See, and
personSunny Rai. year2023.
NewsMet : A `do it all' Dataset of
Contemporary Metaphors in News Headlines. In
booktitleFindings of the Association for Computational
Linguistics: ACL 2023. publisherACL, addressToronto,
Canada, pages10090–10104.

[Junior et al(2022)]
authorpersonIvaldir de Farias Junior,
personSabrina Marczak, personRodrigo Santos,
personCleyton Rodrigues, and personHermano Moura.
C2M: a maturity model for the evaluation of
communication in distributed software development.
journalEmpirical Software Engineering
volume27, number7 (year2022),

[Ke et al(2021)]
authorpersonZixuan Ke, personHu Xu, and
personBing Liu. year2021.
Adapting BERT for Continual Learning of a
Sequence of Aspect Sentiment Classification Tasks. In
booktitleProceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies. publisherACL, addressOnline,

[Kocoń et al(2023)]
authorpersonJan Kocoń, personIgor
Cichecki, personOliwier Kaszyca, personMateusz
Kochanek, personDominika Szydło, personJoanna
Baran, personJulita Bielaniewicz, personMarcin
Gruza, personArkadiusz Janz, personKamil Kanclerz,
personAnna Kocoń, personBartłomiej Koptyra,
personWiktoria Mieleszczenko-Kowszewicz, personPiotr
Miłkowski, personMarcin Oleksy, personMaciej
Piasecki, personŁukasz Radliński, personKonrad
Wojtasik, personStanisław Woźniak, and
personPrzemysław Kazienko. year2023.
ChatGPT: Jack of all trades, master of none.
journalInformation Fusion  volume99
(year2023), pages101861.

authorpersonZ. Kövecses.
Emotion concepts: Social constructionism and
cognitive linguistics.
In booktitleThe verbal communication of emotions.
publisherPsychology Press, pages117–132.

[Kronrod and Danziger(2013)]
authorpersonAnn Kronrod and personShai
Danziger. year2013.
“Wii Will Rock You!” The Use and Effect of
Figurative Language in Consumer Reviews of Hedonic and Utilitarian
journalJournal of Consumer Research
volume40, number4 (date07
year2013), pages726–739.

[Lan et al(2019)]
authorpersonZhenzhong Lan, personMingda
Chen, personSebastian Goodman, personKevin Gimpel,
personPiyush Sharma, and personRadu Soricut.
Albert: A lite bert for self-supervised learning of
language representations.
journalArXiv  volumeabs/1909.11942

[Lin et al(2022)]
authorpersonBin Lin, personNathan Cassee,
personAlexander Serebrenik, personGabriele Bavota,
personNicole Novielli, and personMichele Lanza.
Opinion Mining for Software Development: A
Systematic Literature Review.
journalACM Trans. Softw. Eng. Methodol.
volume31, number3, Article articleno38
(datemar year2022), numpages41 pages.

[Lin et al(2018)]
authorpersonBin Lin, personFiorella
Zampetti, personGabriele Bavota, personMassimiliano
Di Penta, personMichele Lanza, and personRocco
Oliveto. year2018.
Sentiment Analysis for Software Engineering: How
Far Can We Go?. In booktitleProceedings of the 40th ICSE.
publisherACM, addressUSA, pages94–104.

authorpersonBing Liu. year2015.
booktitleSentiment Analysis - Mining Opinions,
Sentiments, and Emotions.
publisherCambridge University Press,
addressCambridge, United Kingdom.

[Liu et al(2022)]
authorpersonEmmy Liu, personChenxuan Cui,
personKenneth Zheng, and personGraham Neubig.
Testing the Ability of Language Models to Interpret
Figurative Language. In booktitleProceedings of the 2022
Conference of the NAACL: Human Language Technologies.
publisherACL, addressSeattle, United States,

[Liu et al(2019)]
authorpersonYinhan Liu, personMyle Ott,
personNaman Goyal, personJingfei Du,
personMandar Joshi, personDanqi Chen,
personOmer Levy, personMike Lewis,
personLuke Zettlemoyer, and personVeselin
Stoyanov. year2019.
Roberta: A robustly optimized bert pretraining
journalArXiv  volumeabs/1907.11692

[Liu et al(2020)]
authorpersonZhen Liu, personShao-hui Lei,
personYu-lang Guo, and personZhi-ang Zhou.
The interaction effect of online review language
style and product type on consumers' purchase intentions.
journalPalgrave Communications
volume6, number1 (year2020),

[Mao et al(2023)]
authorpersonRui Mao, personQian Liu,
personKai He, personWei Li, and
personErik Cambria. year2023.
The Biases of Pre-Trained Language Models: An
Empirical Study on Prompt-Based Sentiment Analysis and Emotion Detection.
journalIEEE Trans. Affect. Comput.
volume14, number3 (year2023),

authorpersonJames H Martin.
A corpus-based analysis of context effects on
metaphor comprehension.
journalTrends in Linguistics Studies and
Monographs  volume171 (year2006),

authorpersonMatthew S. McGlone.
Conceptual Metaphors and Figurative Language
Interpretation: Food for Thought?
journalJournal of Memory and Language
volume35, number4 (year1996),

[Mohammad et al(2016)]
authorpersonSaif Mohammad, personEkaterina
Shutova, and personPeter Turney.
Metaphor as a Medium for Emotion: An Empirical
Study. In booktitleProceedings of the Fifth Joint
Conference on Lexical and Computational Semantics.
publisherACL, addressGermany,

authorpersonSaif M. Mohammad.
9 - Sentiment Analysis: Detecting Valence,
Emotions, and Other Affectual States from Text.
In booktitleEmotion Measurement,
editorpersonHerbert L. Meiselman (Ed.).
publisherWoodhead Publishing, addressUnited Kingdom,

[Munezero et al(2014)]
authorpersonM. Munezero, personC. Montero,
personE. Sutinen, and personJ. Pajunen.
Are They Different? Affect, Feeling, Emotion,
Sentiment, and Opinion Detection in Text.
journalIEEE Transactions on Affective Computing
volume5, number02 (dateapr
year2014), pages101–111.

[Murgia et al(2014)]
authorpersonAlessandro Murgia,
personParastou Tourani, personBram Adams, and
personMarco Ortu. year2014.
Do Developers Feel Emotions? An Exploratory
Analysis of Emotions in Software Artifacts. In
booktitleProceedings of the 11th Working Conference on
MSR. publisherACM, addressUSA,

[Naseem et al(2022)]
authorpersonUsman Naseem, personJinman Kim,
personMatloob Khushi, and personAdam G Dunn.
Robust Identification of Figurative Language in
Personal Health Mentions on Twitter.
journalIEEE Transactions on Artificial
Intelligence volume4, number2
(year2022), pages362–372.

[Nie et al(2020)]
authorpersonYixin Nie, personAdina
Williams, personEmily Dinan, personMohit Bansal,
personJason Weston, and personDouwe Kiela.
Adversarial NLI: A New Benchmark for Natural
Language Understanding. In booktitleProceedings of the 58th
Annual Meeting of the ACL. publisherACL,
addressOnline, pages4885–4901.

[Novielli et al(2020)]
authorpersonN. Novielli, personF. Calefato,
personD. Dongiovanni, personD. Girardi, and
personF. Lanubile. year2020.
Can We Use SE-specific Sentiment Analysis Tools in
a Cross-Platform Setting?. In booktitle2020 IEEE/ACM 17th
International Conference on MSR. publisherIEEE Computer
Society, addressUSA, pages158–168.

[Novielli et al(2018a)]
authorpersonNicole Novielli, personFabio
Calefato, and personFilippo Lanubile.
A Gold Standard for Emotion Annotation in Stack
Overflow. In booktitleProceedings of the 15th International
Conference on Mining Software Repositories. publisherACM,
addressUSA, pages14–17.

[Novielli et al(2021)]
authorpersonNicole Novielli, personFabio
Calefato, personFilippo Lanubile, and
personAlexander Serebrenik. year2021.
Assessment of Off-the-Shelf SE-Specific Sentiment
Analysis Tools: An Extended Replication Study.
journalEmpirical Softw. Engg.
volume26, number4 (datejul
year2021), numpages29 pages.

[Novielli et al(2018b)]
authorpersonNicole Novielli, personDaniela
Girardi, and personFilippo Lanubile.
A Benchmark Study on Sentiment Analysis for
Software Engineering Research. In booktitleProceedings of
the 15th International Conference on MSR. publisherACM,
addressUSA, pages364–375.

[Oord et al(2018)]
authorpersonAaron van den Oord, personYazhe
Li, and personOriol Vinyals. year2018.
Representation learning with contrastive predictive
journalArXiv  volumeabs/1807.03748

authorpersonOpenAI. year2023.
GPT-4 Technical Report.
journalArXiv  volumeabs/2303.08774

[Peng and Feldman(2017)]
authorpersonJing Peng and personAnna
Feldman. year2017.
Automatic Idiom Recognition with Word Embeddings.
In booktitleInformation Management and Big Data.
publisherSpringer International Publishing,
addressCham, pages17–29.

[Raman et al(2020)]
authorpersonNaveen Raman, personMinxuan
Cao, personYulia Tsvetkov, personChristian
Kästner, and personBogdan Vasilescu.
Stress and Burnout in Open Source: Toward Finding,
Understanding, and Mitigating Unhealthy Interactions. In
booktitleProceedings of the ACM/IEEE 42nd ICSE: NIER.
publisherACM, addressUSA, pages57–60.

[Recupero et al(2019)]
authorpersonDiego Reforgiato Recupero,
personMehwish Alam, personDavide Buscaldi,
personAude Grezka, and personFarideh Tavazoee.
Frame-Based Detection of Figurative Language in
Tweets [Application Notes].
journalIEEE Comput. Intell. Mag.
volume14, number4 (year2019),

[Reimers and Gurevych(2019)]
authorpersonNils Reimers and personIryna
Gurevych. year2019.
Sentence-BERT: Sentence Embeddings using
Siamese BERT-Networks. In booktitleProceedings of the
2019 Conference on EMNLP-IJCNLP. publisherACL,
addressHong Kong, China, pages3982–3992.

[Sajadi et al(2023)]
authorpersonA. Sajadi, personK. Damevski,
and personP. Chatterjee. year2023.
Interpersonal Trust in OSS: Exploring Dimensions of
Trust in GitHub Pull Requests. In booktitle2023 IEEE/ACM
45th International Conference on Software Engineering: NIER.
publisherIEEE Computer Society, addressUSA,

authorpersonJaydeb Sarker.
Identification and Mitigation of Toxic
Communications Among Open Source Software Developers. In
booktitleProceedings of the 37th ICSE.
publisherACM, addressUSA, Article
articleno124, numpages5 pages.

[Sarker et al(2020)]
authorpersonJ. Sarker, personA. Turzo,
and personA. Bosu. year2020.
A Benchmark Study of the Contemporary Toxicity
Detectors on Software Engineering Interactions. In
booktitle2020 27th Asia-Pacific Software Engineering
Conference (APSEC). publisherIEEE Computer Society,
addressUSA, pages218–227.

[Sghaier and Sahraoui(2023)]
authorpersonO. Sghaier and personH.
Sahraoui. year2023.
A Multi-Step Learning Approach to Assist Code
Review. In booktitle2023 IEEE International Conference on
SANER. publisherIEEE Computer Society, addressUSA,

authorpersonEkaterina Shutova.
Models of Metaphor in NLP. In
booktitleProceedings of the 48th Annual Meeting of the
ACL. publisherACL, addressUSA,

[Song et al(2021)]
authorpersonWei Song, personShuhui Zhou,
personRuiji Fu, personTing Liu, and
personLizhen Liu. year2021.
Verb Metaphor Detection via Contextual Relation
Learning. In booktitleProceedings of the 59th Annual
Meeting of the ACL and the 11th IJCNLP. publisherACL,
addressOnline, pages4240–4251.

authorpersonGuy Lewis Steele.
Macaroni is Better than Spaghetti. In
booktitleProceedings of the 1977 Symposium on Artificial
Intelligence and Programming Languages. publisherACM,
addressUSA, pages60–66.

[Stowe et al(2022)]
authorpersonKevin Stowe, personPrasetya
Utama, and personIryna Gurevych.
IMPLI: Investigating NLI Models' Performance
on Figurative Language. In booktitleProceedings of the 60th
Annual Meeting of the ACL. publisherACL,
addressDublin, Ireland, pages5375–5388.

[Su et al(2020)]
authorpersonChuandong Su, personFumiyo
Fukumoto, personXiaoxi Huang, personJiyi Li,
personRongbo Wang, and personZhiqun Chen.
DeepMet: A Reading Comprehension Paradigm for
Token-level Metaphor Detection. In booktitleProceedings of
the Second Workshop on Figurative Language Processing.
publisherACL, addressOnline, pages30–39.

[Suhaimi et al(2023)]
authorpersonNur Suhailayani Suhaimi,
personZalinda Othman, and personMohd Ridzwan
Yaakub. year2023.
Comparative Analysis Between Macro and
Micro-Accuracy in Imbalance Dataset for Movie Review Classification. In
booktitleProceedings of Seventh International Congress on
Information and Communication Technology. publisherSpringer
Nature Singapore, addressSingapore, pages83–93.

[Sun et al(2019)]
authorpersonChi Sun, personXipeng Qiu,
personYige Xu, and personXuanjing Huang.
How to Fine-Tune BERT for Text Classification?. In
booktitleChinese Computational Linguistics.
publisherSpringer International Publishing,
addressCham, pages194–206.

[Tian et al(2013)]
authorpersonYuan Tian, personD. Lo, and
personChengnian Sun. year2013.
DRONE: Predicting Priority of Reported Bugs by
Multi-factor Analysis. In booktitle2013 IEEE International
Conference on Software Maintenance (ICSM). publisherIEEE
Computer Society, addressUSA, pages200–209.

[Tian et al(2015)]
authorpersonYuan Tian, personDavid Lo,
personXin Xia, and personChengnian Sun.
Automated prediction of bug report priority using
multi-factor analysis.
journalEmpirical Software Engineering
volume20, number5 (date01 10
year2015), pages1354–1383.

[Tiwari et al(2023)]
authorpersonDeepika Tiwari, personTim
Toady, personMartin Monperrus, and personBenoit
Baudry. year2023.
With Great Humor Comes Great Developer Engagement.
journalArXiv  volumeabs/2312.01680

[Umer et al(2019)]
authorpersonQasim Umer, personHui Liu,
and personInam Illahi. year2019.
CNN-based automatic prioritization of bug reports.
journalIEEE Transactions on Reliability
volume69, number4 (year2019),

[Umer et al(2018)]
authorpersonQasim Umer, personHui Liu,
and personYasir Sultan. year2018.
Emotion based automated priority prediction for bug
journalIEEE Access  volume6
(year2018), pages35743–35752.

[Veale and Hao(2008)]
authorpersonTony Veale and personYanfen
Hao. year2008.
A Fluid Knowledge Representation for Understanding
and Generating Creative Metaphors. In booktitleProceedings
of the 22nd International Conference on Computational Linguistics (Coling
2008). publisherColing 2008 Organizing Committee,
addressManchester, UK, pages945–952.

[Wang et al(2023)]
authorpersonWen-Yao Wang, personChen-Hao
Wu, and personJie He. year2023.
CLeBPI: Contrastive Learning for Bug Priority
journalInformation and Software Technology
volume164 (year2023), pages107302.

[Weitzel et al(2016)]
authorpersonLeila Weitzel,
personRonaldo Cristiano Prati, and personRaul Freire
Aguiar. year2016.
booktitleThe Comprehension of Figurative Language:
What Is the Influence of Irony and Sarcasm on NLP Techniques?
publisherSpringer International Publishing,
addressCham, pages49–74.

[Wijesiriwardene et al(2023)]
authorpersonThilini Wijesiriwardene,
personAmit Sheth, personValerie L Shalin, and
personAmitava Das. year2023.
Why Do We Need Neurosymbolic AI to Model Pragmatic
journalIEEE Intelligent Systems
volume38, number5 (year2023),

[Yan et al(2022)]
authorpersonHanqi Yan, personLin Gui,
personWenjie Li, and personYulan He.
Addressing token uniformity in transformers via
singular value transformation. In booktitleProceedings of
the 38th Conference on Uncertainty in Artificial Intelligence,
Vol. volume180. publisherPMLR,
addressEindhoven, The Netherlands, pages2181–2191.

[Zhang et al(2012)]
authorpersonLei Zhang, personYanchun Sun,
personHui Song, personWeihu Wang, and
personGang Huang. year2012.
Detecting Anti-Patterns in Java EE Runtime System
Model. In booktitleProceedings of the Fourth Asia-Pacific
Symposium on Internetware. publisherACM,
addressUSA, Article articleno21,
numpages8 pages.

[Zhang et al(2020)]
authorpersonT. Zhang, personB. Xu,
personF. Thung, personS. Haryono, personD.
Lo, and personL. Jiang. year2020.
Sentiment Analysis for Software Engineering: How
Far Can Pre-trained Transformer Models Go?. In
booktitle2020 IEEE International Conference on Software
Maintenance and Evolution (ICSME). publisherIEEE Computer
Society, addressUSA, pages70–80.

[Zhou et al(2021)]
authorpersonJianing Zhou, personHongyu
Gong, and personSuma Bhat. year2021.
PIE: A Parallel Idiomatic Expression Corpus for
Idiomatic Sentence Generation and Paraphrasing. In
booktitleProceedings of the 17th Workshop on Multiword
Expressions (MWE 2021). publisherACL,
addressOnline, pages33–48.