# AStitchInLanguageModels: Dataset and Methods for the Exploration of
Idiomaticity in Pre-Trained Language Models

Harish Tayyar Madabushi, Edward Gow-Smith,
Carolina Scarton Aline Villavicencio
Department of Computer Science, University of Sheffield
United Kingdom
{h.tayyarmadabushi, egow-smith1, c.scarton, a.villavicencio}
@sheffield.ac.uk

###### Abstract

Despite their success in a variety of NLP tasks, pre-trained language models,
due to their heavy reliance on compositionality, fail in effectively capturing
the meanings of multiword expressions (MWEs), especially idioms. Therefore,
datasets and methods to improve the representation of MWEs are urgently
needed. Existing datasets are limited to providing the degree of idiomaticity
of expressions along with the literal and, where applicable, (a single) non-
literal interpretation of MWEs. This work presents a novel dataset of
naturally occurring sentences containing MWEs manually classified into a fine-
grained set of meanings, spanning both English and Portuguese. We use this
dataset in two tasks designed to test i) a language model’s ability to detect
idiom usage, and ii) the effectiveness of a language model in generating
representations of sentences containing idioms. Our experiments demonstrate
that, on the task of detecting idiomatic usage, these models perform
reasonably well in the one-shot and few-shot scenarios, but that there is
significant scope for improvement in the zero-shot scenario. On the task of
representing idiomaticity, we find that pre-training is not always effective,
while fine-tuning could provide a sample efficient method of learning
representations of sentences containing MWEs.

##  1 Introduction and Motivation

Pre-trained language models such as BERT Devlin et al. (2019) and XLNet Yang
et al. (2019) have been widely used in a variety of Natural Language
Processing tasks. Despite their success in multiple downstream applications,
such as sentence classification Zhang et al. (2019) and reading comprehension
Raffel et al. (2019), they are unable to effectively represent idiomatic
multiword expressions (MWEs) Yu and Ettinger (2020); Garcia et al. (2021).
Capturing idiomaticity is particularly challenging as the representations of
words and phrases are explicitly designed to be compositional both in non-
contextual Mitchell and Lapata (2010); Mikolov et al. (2013b) and contextual
embedding models. Pre-trained language models in particular exploit
compositionality at both the word and sub-word levels Devlin et al. (2019) to
reduce the size of their vocabulary, which makes representing idiomatic
phrases particularly challenging. The effective representation of idiomatic
MWEs is critical for them to be correctly interpreted in downstream tasks.
Such an improvement will benefit both classification-based problems (e.g.
sentiment analysis) and sequence-to-sequence tasks (e.g. machine translation).

To this end, we present a dataset consisting of naturally occurring sentences
containing potentially idiomatic MWEs and two tasks aimed at evaluating
language models’ ability to effectively detect and represent idiomaticity. The
primary contributions of this work are:

  1. 1.

A novel dataset consisting of:

    1. (a)

naturally occurring sentences (and two surrounding sentences) containing
potentially idiomatic MWEs annotated with a fine-grained set of meanings:
compositional meaning, idiomatic meaning(s), proper noun and “meta usage”;

    2. (b)

paraphrases for each meaning of each MWE;

  2. 2.

Two tasks aiming at evaluating i) a model’s ability to detect idiomatic usage,
and ii) the effectiveness of sentence embeddings in representing idiomaticity.
Table 1 provides details of these tasks and associated subtasks, each designed
to test different aspects of models.

    1. (a)

These tasks are presented in multilingual, zero-shot, one-shot and few-shot
settings.

    2. (b)

We provide strong baselines using state-of-the-art models, including
experiments with one-shot and few-shot setups for idiomaticity detection and
the use of the _idiom principle_ for detecting and representing MWEs in
contextual embeddings. Our results highlight the significant scope for
improvement.

Task 1 |  Subtask A |  Coarse-grained classification of examples containing idioms.
---|---|---
|  Subtask B |  Fine-grained classification of examples into meanings.
Task 2 |  Subtask A |  Effective representation of sentences containing idiomatic phrases using only _pre-training_.
|  Subtask B |  Effective representation of sentences using both pre-training and _fine-tuning_.
Table 1:  AStitchInLanguageModels Tasks: The two tasks and associated
subtasks.

This dataset and associated tasks have the potential to catalyse research into
representing more complex elements of language beginning with idiomaticity,
thus ensuring a timely stitch in language models. We call this dataset and
associated tasks _AStitchInLanguageModels_ , and make the dataset, the
associated splits for each task, pre-training data, pre-trained and fine-tuned
models, program code and associated processing scripts, including
hyperparameters, publicly available in the interest of reproducibility and for
subsequent
reuse111https://github.com/H-TayyarMadabushi/AStitchInLanguageModels.

This paper is organised as follows: Section 2 presents a discussion of related
work. We then present AStitchInLanguageModels consisting of the novel MWE
dataset and the two associated tasks in Section 3. We discuss our experiments
and results for these two tasks in Section 4, before presenting a discussion
of the more interesting elements of our findings in Section 5. We present our
conclusions and possible avenues of future work in Section 6.

##  2 Related work

The problems posed by MWEs to NLP models have been known for some time Sag et
al. (2002); Constant et al. (2017); Shwartz and Dagan (2019). For instance,
Sag et al. (2002) refer to the _idiomaticity problem_ and place the need for
effective processing of MWEs on par with that for word sense disambiguation to
be able to effectively process text. While their analysis focused on symbolic
methods, this problem still persists: Shwartz and Dagan (2019) showed, using
six tasks, that contextual pre-trained language models, capable of handling
polysemy, continued to be unable to effectively handle idiomatic MWEs,
although they tend to do better than their non-contextual predecessors.
Further experiments with probing pre-trained language models across multiple
languages have also confirmed this result Yu and Ettinger (2020); Garcia et
al. (2021).

###  2.1 Existing Datasets

Datasets of MWE annotated corpora include that associated with the PARSEME
shared task Savary et al. (2017) which focuses on verbal MWEs and the STREUSLE
dataset Schneider et al. (2014); Schneider and Smith (2015); Schneider et al.
(2016) which includes noun, verb, prepositional and possessive expressions
including “semantic supersenses”. However, most existing datasets associated
with compositionality of MWEs consist of isolated phrases, labelled with
overall compositionality scores Venkatapathy and Joshi (2005); Biemann and
Giesbrecht (2011); Farahmand et al. (2015), scores of how individual words
contribute to the meaning of the MWE Venkatapathy and Joshi (2005), or both
Reddy et al. (2011); Cordeiro et al. (2019); Schulte im Walde et al. (2016).
While most of these target only English, some include scores for other
languages such as German Schulte im Walde et al. (2016), and French and
Portuguese Cordeiro et al. (2019).

Existing datasets of compositionality that include context often add context
automatically by first selecting MWEs that are either only compositional or
only idiomatic. For instance, the VNC-Tokens Dataset Cook et al. (2008)
consists of 53 English MWEs each with a maximum of 100 sentences extracted
from the BNC, while Tu and Roth (2012) collected 1,348 sentences associated
with 23 verb phrases annotated as compositional and idiomatic. Shwartz and
Dagan (2019) focused on a subset of noun compounds that are only compositional
or idiomatic from the dataset provided by Reddy et al. (2011) and
automatically added sentences from Wikipedia. Finally, the NCS Dataset Garcia
et al. (2021) consists of 280 English and 180 Portuguese MWEs, annotated with
degrees of compositionality and three sentences containing each of the MWEs.

Despite the importance of the context surrounding an MWE, where available,
context, in the form of sentences containing MWEs, is available only for those
MWEs that are either idiomatic or compositional. This significant shortcoming
makes it impossible to train models to learn to differentiate between the
compositional and idiomatic usage of the same MWE.

Finally, while existing datasets also provide paraphrases for the
compositional and idiomatic meanings of MWEs Hendrickx et al. (2013); Garcia
et al. (2021), they are limited to having exactly one compositional and one
idiomatic meaning, which is not always the case as is exemplified by the
phrase “head hunter” which, while not having a literal usage, has multiple
idiomatic meanings (i.e recruiter, baseball pitcher who aims for the head, and
hunter).

_AStitchInLanguageModels_ is designed to alleviate these shortcomings,
specifically: a) the lack of context sentences, b) the need for fine grained
classification of MWEs, and a more complete set of paraphrases for all
possible meanings of MWEs (Section 3).

###  2.2 Methods

The task of identifying idiomaticity in sentences was initially addressed by
use of symbolic methods Baldwin and Villavicencio (2002); Sag et al. (2002),
statistical properties of text such as mutual information Lin (1999), and
latent semantic analysis Baldwin et al. (2003).

The subsequent adoption of distributional semantics led to the use of
constituent word embeddings to determine the compositionality of phrases, such
as in the work by Katz and Giesbrecht (2006) who made use of the semantic
similarity between the distributional vectors associated with an MWE as a
whole and those associated with its parts to determine compositionality. This
is achieved by use of a single token to represent an MWE. This trend continued
with the introduction of neural distributional semantic models such as
word2vec Mikolov et al. (2013a) wherein MWEs were taken as single units in
learning embeddings Mikolov et al. (2013b). This method was improved upon by
use of an explicit disambiguation step prior to composition Kartsaklis et al.
(2014), and by the joint learning of compositional and idiomatic embeddings
using a “compositionality scoring” function Hashimoto and Tsuruoka (2016).
This “single token” method has the advantage of being rooted in the linguistic
_idiom principle_ Sinclair et al. (1991), which postulates that humans process
idioms by treating them as a “single independent token”.

Despite being the only method of handling MWEs and having had relative
success, it is not without its shortcomings. The first is that the frequency
of MWEs tends to be low (a problem that worsens with the increase in length of
MWEs) and since the quality of distributional representations tends be
proportional to the number of instances of a token, representations of MWEs
are often lacking. The second is that non-contextual type level
representations are inherently limited as MWEs often have multiple meanings,
as detailed in Section 2.1.

While contextual embeddings can handle polysemy, they fail to fully capture
the meaning of MWEs as discussed earlier. How contextual embeddings fair in
comparison to their non-contextual predecessors is not entirely clear as
Nandakumar et al. (2019) found that they do worse on some tasks while Shwartz
and Dagan (2019) found that they do better. Hashempour and Villavicencio
(2020) adopted the idiom principle (MWE as a single token) with contextual
language models (specifically BERT), and found that this method does not
benefit transformer-based pre-trained models. However, they did not introduce
a new token to represent each MWE as is required during the training of non-
contextual models built on the idiom principle, but instead replaced MWEs with
a single token in the input and rely on BERT’s word-piece tokenizer. To the
best of our knowledge this work is the first to introduce new tokens for MWEs
into a contextual pre-trained language model (see Section 4.2).

##  3 AStitchInLanguageModels: Dataset and Tasks

To create a dataset and tasks aimed at improving language models’ ability to
identify and capture idiomaticity, we first collected examples of MWE usage in
naturally occurring sentences along with the two surrounding sentences. We
then annotated these examples with a fine-grained set of meanings associated
with each usage. We restrict our attention to noun compounds, a subset of
idiomatic MWEs, sourced from the Noun Compound Senses (NCS) dataset Cordeiro
et al. (2019), which extends the dataset by Reddy et al. (2011).

###  3.1 Data Collection and Annotation

A total of 12 judges were asked to collect examples containing a list of MWEs
occurring naturally in context, in both English and Portuguese. For each MWE,
judges were instructed to obtain 7 to 10 examples of each meaning (
“Idiomatic”, “Non-Idiomatic”, “Proper Noun” and “Meta Usage”) where possible,
with between 20 and 30 total examples for each MWE. We define “Meta Usage” to
be the literal use of an MWE in a metaphor (e.g. life vest in “Let the Word of
God be our life vest to keep us afloat, so as not to drown.”). Judges were
additionally instructed to add to the list of possible meanings associated
with each MWE based on the usage they observed when collecting examples, or to
flag examples with novel usage for review by language experts. Emphasis was
put on extracting high-quality examples with three contiguous sentences and
correct formatting, containing no unusual characters. The data consists of
excerpts of text from the web, each a maximum of three sentences, thus
adhering to fair use.

The meanings of each MWE were then paraphrased by language experts. The
idiomatic paraphrases aim to concisely convey the meaning of the idiom. For
example, _cutting edge_ is paraphrased to _most advanced_ and _night owl_ is
paraphrased to _nocturnal person_ in the idiomatic case. The aim of the
literal paraphrase is to apply a minimal lexical alteration that shifts the
MWE away from its idiomatic meaning(s). For example, _cutting edge_ is
paraphrased to _slicing edge_ and _night owl_ is paraphrased to _night hooter_
in the literal case. This adversarial paraphrasing is designed to test a
model’s ability to discern a compositional meaning from an idiomatic one, and
aims to ensure that models must have a nuanced understanding of idiomaticity
for them to be successful. Examples of the annotated data are shown in Table
2.

MWE |  Target Sentence |  Previous Sentence |  Next Sentence | Label | Idiomatic? | Paraphrase
---|---|---|---|---|---|---
gold mine |  This means that search data is a _gold mine_ for marketing strategy. (marketingweek.com) |  The data that those searches generate builds… |  It reveals which types of product… | Idiomatic 1 | Yes | source of fortune
gold mine |  The hashtag “Qixia _gold mine_ incident” has been viewed many million of times on the social media site Weibo. (wsws.org) |  The rescue operation took place… |  A week after the explosion… | Literal | No | mine
gold mine |  The _Gold Mine_ ’s plain frontage & sparse, white-walled dining room suggest that it’s a quick-fix refuelling stop rather than a place to linger. (squaremeal.co.uk) |  SquareMeal Review of Gold Mine |  The menu touts a bewildering array of dishes… | Proper Noun | No | Proper Noun

Table 2: A sample of the dataset for one MWE (_gold mine_). Context sentences
are truncated for brevity. The "Idiomatic?" column is used for the coarse-
grained classification task (Subtask 1 A) and the "Paraphrase" column is used
for the fine-grained classification task (Subtask 1 B) and the representation
task (Task 2).

Finally, each example was annotated with a label and corresponding paraphrase
by two judges. The Cohen’s kappa coefficient of inter-rater reliability was
0.887 for English and 0.807 for Portuguese. We note that a significant
proportion of disagreements arose from a difference in interpretation of the
“Proper Noun” and “Meta Usage” labels, and from what constituted “low quality”
for discarding examples. For resolution of disagreements a final label was
decided based on a discussion between the judges.

###  3.2 The Final Dataset

The final dataset consists of 4,558 English examples containing 223 MWEs, and
1,872 Portuguese examples containing 113 MWEs.

We divide this data into training, development and test splits as follows: the
test and development splits consist of sentences containing 30 and 20 idioms
each in English and Portuguese respectively. To enable the testing of models
under different scenarios of data availability, we create three different
setups of the test split for each language. The first, the _zero-shot_ setup,
consists of sentences containing 163 idioms in English and 60 idioms in
Portuguese, which do not occur in the development and test sets. The second,
the _one-shot_ setup, consists of exactly one non-idiomatic and (where
available) one idiomatic example associated with each MWE in the development
and test sets. The third and final, the _few-shot_ setup, consists of between
1 and 4 examples associated with each meaning of each MWE in the development
and test sets. The exact number of examples available is proportional to the
original number of examples associated with that specific meaning of that
idiom. We make it clear that there are no overlapping target sentences between
the three splits - the only overlap is in terms of the idioms contained in
examples. Detailed statistics for the English and Portuguese datasets are
provided in Appendix A.

###  3.3 Tasks

In addition to the dataset of labelled contextualised MWEs, we present two
tasks.

####  3.3.1 Task 1: Idiomaticity Detection

The first task we propose is designed to evaluate the extent to which models
can identify idiomaticity in text and consists of two Subtasks: a _coarse-
grained_ classification task (Subtask A) and a _fine-grained_ classification
task (Subtask B). For the coarse-grained subtask, the problem is simplified to
classifying the examples as either “Idiomatic” or “Non-Idiomatic”. For the
purposes of this subtask, anything labelled as “Literal” or “Proper Noun” was
classed as “Non-Idiomatic” and given a label of 1, whilst all “Idiomatic”
labels as well as “Meta Usage” were given a label of 0. (See also Table 2).

For the fine-grained task, the possible meanings are equivalent to the
paraphrases in the dataset, described previously. Since this problem does not
have a fixed number of labels (given that each MWE has a different set of
meanings), we convert this to a binary classification problem: the first input
is the example containing the MWE and the second the paraphrase of each
possible meaning of the MWE (or one of the phrases “Proper Noun” or “Meta
Usage”). An input pair is labelled 1 if the paraphrase represents the correct
meaning of the MWE in the example and 0 otherwise. In addition, we report
scores for both subtasks in the zero-shot, one-shot and few-shot setups to
better evaluate a model’s ability to generalise and learn in a sample
efficient fashion. We note that this was impossible prior to the introduction
of AStitchInLanguageModels as all previous datasets which included context
considered only one meaning per MWE (see Section 2.1). Due to the imbalanced
nature of these subtasks, we use Macro F1-score as the measure of evaluation.

We note that due to the different ways in which the two settings in this Task
are setup the results for the two settings in this task are not directly
comparable.

Sentence (E) |  Correct Replacement (  $E_{\text{MWE}\rightarrow\text{c}}$ ) |  Wrong Replacement (  $E_{\text{MWE}\rightarrow\text{i}}$ ) |  Expected
---|---|---|---
When removing a _big fish_ from a net, it should be held in a manner that supports the girth. (newsdakota.com) |  When removing a _fish_ from a net, it should be held in a manner that supports the girth. |  When removing a _important person_ from a net, it should be held in a manner that supports the girth. |  | $sim(E,E_{\rightarrow\text{c}})=1$
---
$sim(E,E_{\rightarrow\text{i}})=sim(E_{\rightarrow\text{c}},E_{\rightarrow\text{i}})$

To pay attention only to new housing and houses I think skews the _big picture_. (streets.mn) |  To pay attention only to new housing and houses I think skews the _whole situation_. |  To pay attention only to new housing and houses I think skews the _large image_. |  | $sim(E,E_{\rightarrow\text{c}})=1$
---
$sim(E,E_{\rightarrow\text{i}})=sim(E_{\rightarrow\text{c}},E_{\rightarrow\text{i}})$

Table 3:  Task 2 - Models are required to be consistent in assigning semantic
similarity scores as measured by use of the paraphrases of different meanings.

####  3.3.2 Task 2: Idiomaticity Representation

While the identification of idiomaticity is important, downstream tasks
require embeddings that effectively capture idiomaticity, which is the purpose
of the second task. For this task, we design a metric to measure how
consistent a model is in capturing similarity between sentences containing
idiomatic elements and sentences that are purely compositional.

As each possible meaning of an MWE contained in each example is associated
with a paraphrase, this task requires a model to generate similarity scores
for each example $E$ such that:

$\begin{split}\forall_{i\in I}\Big{(}&sim(E,E_{\rightarrow\text{c}})=1;\\\
&sim(E,E_{\rightarrow\text{i}})=sim(E_{\rightarrow\text{c}},E_{\rightarrow\text{i}})\Big{)}\end{split}$
(1)

where $E_{\rightarrow\text{c}}$ represents the example with the MWE in $E$
replaced by the paraphrase of the correct meaning associated with the MWE, and
$E_{\rightarrow\text{i}}$ the example with the MWE replaced by a paraphrase of
one of the incorrect meanings of the MWE in $E$ (see Table 3 for examples).

Without additional checks, models can trivially succeed in this task by simply
assigning a similarity score of 1 to every sentence pair. To prevent this, we
splice in development and test data from the Semantic Text Similarity (STS)
benchmark dataset Cer et al. (2017) in English and from the ASSIN2 STS dataset
Real et al. (2020) for Portuguese.

We note that the expected similarity scores are approximates as the
paraphrases need not have _exactly_ the same meanings as that of the MWE they
are paraphrasing. However, we consider this difference to be acceptable given
the typical nature of annotation of semantic similarity data wherein
annotators use labels between 1 and 5.

Finally, we divide this task into two subtasks: Subtask A which requires the
solving of this task using _only pre-training_ and Subtask B which allows the
_fine-tuning_ of models. For clarity, we define pre-training to be the
training of a model on any task _other than idiomatic STS_ (and can include
“fine-tuning” on a different task), and fine-tuning to include the inclusion
of training on any STS dataset which includes potentially idiomatic MWEs. We
use Spearman correlation coefficient as the measure of evaluation for both
subtasks in Task 2 as it has been shown that Pearson correlation is poorly
suited for comparing performance on the STS task Reimers et al. (2016).

##  4 Experiments and Results

Our aim was to investigate the performance of state-of-the-art transformer-
based pre-trained language models on these tasks, and how their performance
varied with different input features (i.e. inclusion of MWE, context
sentences), problem setups (i.e. zero-shot, one-shot and few-shot), and
training regimes (i.e. pre-training, fine-tuning) so as to provide a baseline
for the AStitchInLanguageModels dataset. Here we provide an overview of the
experiments ran and our results. More detailed description of the experimental
procedure, including runtimes are given in Appendix B.

###  4.1 Task 1: Idiomaticity Detection

For Subtask A, which requires the coarse-grained classification of examples,
we start by exploring the impact of three variables in the zero-shot setup:
the pre-trained language model, the inclusion of context (the two surrounding
sentences), and adding the relevant MWE as a feature. The context is included
by simply concatenating the three contiguous sentences, and the MWE is
included by separating it from the rest of the input by use of the “[SEP]”
token. For the purposes of brevity, we report a subset of variations
highlighting the most interesting results, with more details of the
experimental procedure in Appendix B. Among the results for Task 1, Subtask A
(Table 4), the best-performing experimental settings from the zero-shot
setting (by development F1 score) were transferred over to the one-shot and
few-shot problem setups. While the inclusion of context (surrounding
sentences) did not change the performance of the models significantly, and
will not be used in the other experiments, the inclusion of the relevant MWE
was found to be beneficial to performance.

For Subtask B, fine-grained classification, the best-performing experimental
settings found for the first subtask were used for the multiclass data,
although the MWE was not included as a feature, since our previous method for
inclusion is incompatible with the passing of the paraphrase; the input
consists of the target sentence without the previous or next sentences
followed by a single possible meaning of the MWE separated by the “[SEP]”
token. The task is thus reduced to a binary classification task wherein the
model is required to predict 1 when the target sentence is followed by the
correct paraphrase and 0 otherwise. The results are in Table 5.

| Problem Setup | Model | Context? | MWE? | Dev F1 | Test F1
---|---|---|---|---|---|---
English | zero-shot | BERT base (cased) | No | No | 0.724 | 0.688
BERT base (cased) | Yes | No | 0.717 | 0.797
BERT base (cased) | Yes | Yes | 0.779 | 0.774
BERT base (cased) | No | Yes | 0.785 | 0.821
| XLNet base (cased) | No | Yes | 0.823 | 0.832
one-shot | XLNet base (cased) | No | Yes | 0.897 | 0.874
| one-shot | XLNet base (cased) | Yes | No | 0.689 | 0.701
| one-shot | XLNet base (cased) | No | No | 0.755 | 0.754
| few-shot | XLNet base (cased) | No | Yes | 0.959 | 0.971
| few-shot | XLNet base (cased) | Yes | No | 0.782 | 0.806
| few-shot | XLNet base (cased) | No | No | 0.792 | 0.853
Portuguese | zero-shot | XLM-RoBERTa base (cased) | No | No | 0.593 | 0.528
XLM-RoBERTa base (cased) | Yes | No | 0.542 | 0.562
XLM-RoBERTa base (cased) | Yes | Yes | 0.696 | 0.604
XLM-RoBERTa base (cased) | No | Yes | 0.703 | 0.579
| BERT base multilingual (cased) | No | Yes | 0.686 | 0.560
one-shot | XLM-RoBERTa base (cased) | No | Yes | 0.877 | 0.778
| one-shot | XLM-RoBERTa base (cased) | Yes | No | 0.605 | 0.563
| one-shot | XLM-RoBERTa base (cased) | No | No | 0.638 | 0.534
| few-shot | XLM-RoBERTa base (cased) | No | Yes | 0.926 | 0.944
| few-shot | XLM-RoBERTa base (cased) | Yes | No | 0.655 | 0.684
| few-shot | XLM-RoBERTa base (cased) | No | No | 0.796 | 0.696
Table 4: Evaluation results for Task 1 Subtask A (with best results for each setting in bold). | Problem Setup | Model | Dev F1 | Test F1
---|---|---|---|---
En | zero-shot | XLNet base (cased) | 0.852 | 0.875
one-shot | XLNet base (cased) | 0.923 | 0.927
few-shot | XLNet base (cased) | 0.933 | 0.948
Pt | zero-shot | XLM-RoBERTa base (cased) | 0.843 | 0.778
one-shot | XLM-RoBERTa base (cased) | 0.852 | 0.858
few-shot | XLM-RoBERTa base (cased) | 0.909 | 0.878
Table 5: Evaluation results for Task 1 Subtask B.

###  4.2 Task 2: Idiomaticity Representation

Task 2 requires models to output the semantic similarity between sentences in
a consistent manner. Given that sentence embeddings generated by pre-trained
language models cannot directly be used to calculate semantic similarity
Devlin et al. (2019), we used Sentence BERT Reimers and Gurevych (2019) which
consists of a siamese network structure with a regression objective function
consisting of the mean-squared error loss calculated over the cosine
similarity of two input sentences during training. This results in sentences
whose semantic similarity can be compared using cosine similarity Schroff et
al. (2015). We note that while this is not strictly required for our purpose,
we use this method as the siamese network structure is likely to be beneficial
in fine-tuning on the idiomatic STS data where the similarity scores are all
relatively close to each other.

To test the effectiveness of the idiom principle to represent MWEs (Section
2.2) for Task 2, we analyse three different settings, involving the expansion
of the vocabulary of pre-trained models by the addition of a single token to
represent each MWE: In the first setting (“all replace”) all instances of an
MWE are replaced with the corresponding token before input to the model; in
the second (“select replace”) each input sentence is first classified using
the one-shot model for course grained classification (Section 4.1) and a given
instance of an MWE is replaced only when the one-shot model predicts that the
MWE in a given sentence has an idiomatic meaning; and in the third (“no
replace”) there is no change to either the model (no special token added) or
their input.

For Subtask A, which requires the use of only pre-training, we collect
sentences (including, where available, the paragraph they occur in) from the
Common Crawl News Dataset222https://commoncrawl.org spanning the first 6
months of 2020 (over half a terabyte of text). This results in about 220,000
sentences in English and about 16,000 in Portuguese containing relevant MWEs.
We use this data to continue pre-training BERT base in both the “all replace”
and “select replace” variations described above. Unlike our other experiments,
we do not pre-train multiple times due to time and resource constraints. We
also limit pre-training to 5 epochs for English and 10 epochs for Portuguese
based on results from our exploratory experiments.

In addition to these two models, we also test BERT base with no modifications,
and a version of BERT base with the addition of tokens associated with each
MWE but no pre-training (the embeddings associated with these tokens are
randomly initialised). The “all replace” and “select replace” models have
their pre-training and input sentences tokenized according to the same
strategy. Each of these models are subsequently trained using the Sentence
BERT architecture so as to ensure that the resultant embeddings can be
compared using cosine similarity. We train using the training data from the
STS benchmark dataset Cer et al. (2017) for English and the ASSIN2 STS dataset
Real et al. (2020) for Portuguese. This training does _not_ violate the “pre-
train only” requirement of this task as we do not train on idiomatic STS data.
The results are presented in Table 6.

|  Tokenization |  Dev $\rho$ |  Test $\rho$
---|---|---|---
English |  Default |  0.767 |  0.744
|  All Tokenized (No Pre-Training) |  0.826 |  0.801
|  All Tokenized |  0.835 |  0.811
|  Select Tokenized |  0.848 |  0.805
Portuguese |  Default |  0.726 |  0.785
|  All Tokenized (No Pre-Training) |  0.749 |  0.798
|  All Tokenized |  0.742 |  0.805
|  Select Tokenized |  0.750 |  0.814
Table 6:  Results for Task 2 Subtask A.

For Subtask B, we fine-tune the “no replace”, “all replace” and “select
replace” versions of BERT base on both the standard STS data as in Subtask A
and training data constructed from the zero-shot and few-shot version of the
training data using Equation 1. Therefore, during fine-tuning, the gold
similarity score for $sim(E,E_{\rightarrow\text{c}})$ is $1$ and that for
$sim(E,E_{\rightarrow\text{i}})$ is
$sim(E_{\rightarrow\text{c}},E_{\rightarrow\text{i}})$. Both the “replace”
versions of the model are fine-tuned from scratch (i.e. the tokens associated
with MWE are random and not pre-trained as in Subtask A). Although it is
possible to start with the pre-trained version of the “replace” models, we
make the conscious decision not to, so we might test if this sample efficient
method of learning is feasible. The results for these models are presented in
Table 7.

|  Tokenization |  Dev $\rho$ |  Test $\rho$
---|---|---|---
En |  Default |  0.818 |  0.823
|  All Tokenized |  0.821 |  0.817
|  Select Tokenized |  0.851 |  0.825
Pt |  Default |  0.752 |  0.811
|  All Tokenized |  0.803 |  0.835
|  Select Tokenized |  0.806 |  0.818
Table 7:  Results for Task 2 Subtask B.

##  5 Discussion

This section discusses some highlights of our results.

###  5.1 Detection of Idiomaticity

In the task of detecting idiomaticity (Task 1), we find that in the zero-shot
setting, the models perform poorly in both the coarse-grained and fine-grained
subtasks. This shows there is still significant room for improvement in this
task. The most interesting result was that models perform surprisingly well in
the one-shot and few-shot setups. This is a novel observation, made possible
by the unique nature of this dataset and is likely to be very helpful in
developing methods of identifying idiomatic language.

We found that including context sentences did not always lead to significantly
improved model performance. Intuitively, one would expect an increase in
performance due to the availability of more relevant data. A possible reason
we did not observe this in our experiments is that we included context by
simply concatenating the three sentences, which means the model has no
awareness of which sentence is relevant and could be deceived by surrounding
sentences containing idiomatic expressions, for example. However, in the zero
shot setting, including the context while excluding the target MWE led to a
significant increase in generalisability as measured by the increased
performance on the test set. This combination led to an increase of almost 8
points over the development set in English and 2 points in Portuguese where
all other combinations led to a drop in performance on the test set as
compared to the development set.

The inclusion of the relevant MWE, was generally found to be greatly
beneficial to model performance. The intuition behind this is that models are
able to “focus” on the relevant MWE when determining idiomaticity. In the one
and few shot settings in particular, this inclusion significantly boosted
performance. When models had previously not encountered examples associated
with a particular MWE in the training data (as in the zero shot setting),
including the MWE did less to boost performance, although it still did improve
results. The only advantage of excluding the MWE was in helping with
generalisation as detailed above. In the case of both MWE and context
inclusion, we expect more sophisticated methods of incorporating this
information to further boost performance.

We note that results in English outperform those in Portuguese. We believe
that this difference could be a result of three factors: a) the fact that
there is less training data available in Portuguese, b) because models are
pre-trained on significantly less Portuguese data, and c) due to the higher
degree of inflection in Portuguese.

###  5.2 Representation of Idiomaticity

Recall that the evaluation data for Task 2 included data from standard STS
datasets to ensure that the task is not trivially solvable (Section 3.3.2). We
report results on only the MWE subset of the evaluation data in Tables 8 and 9
for Subtasks A and B respectively.

Tokenization |  EN Non-STS $\rho$ |  PT Non-STS $\rho$
---|---|---
Default |  0.219 |  0.203
All Tokenized (No Pre-Training) |  0.395 |  0.274
All Tokenized |  0.459 |  0.369
Select Tokenized |  0.437 |  0.332
Table 8:  Results on only the MWE subset of the Test split for Task 2 Subtask A. Tokenization |  EN Non-STS $\rho$ |  PT Non-STS $\rho$
---|---|---
Default |  0.627 |  0.312
All Tokenized |  0.611 |  0.379
Select Tokenized |  0.618 |  0.416
Table 9: Results on only the MWE subset of the Test split for Task 2 Subtask
B.

These results show the significant scope for improvement in representing
idiomaticity (given model performance on the standard STS benchmark datasets
is close 0.9$\rho$). Additionally, we note that in Subtask A, which requires
the use of only pre-training, it is better to tokenize all pre-training data,
thus maximising the amount of training data, rather than selectively
tokenizing training data. In Subtask B (fine-tuning), however, selective
tokenization seems to have a slight advantage although the default
tokenization seems to be more suitable in English.

Thus, our experiments exploring the use of the idiomatic principle to capture
idiomaticity in contextual pre-trained models (Task 2 Subtask A), show that
while replacing potential MWEs with a single token does improve performance,
further pre-training with text tokenized either using “all replace” or “select
replace” improves performance only on the MWE subset of the evaluation split.
On the full test set, which includes standard STS data (Table 6), however,
additional pre-training with MWE data does not always improve over a random
representation of MWE tokens and when it does, it does so only slightly. This
is an interesting result, and could be because gains made by the use of the
idiom principle are offset by the continuing to pre-train on a relatively
small set of sentences that include a randomly initialised token added to the
vocabulary, or because the gains made on the MWE subset are diluted across the
entire test split.

Experiments using fine-tuning (Task 2 Subtask B, see Section 4.2, Table 6)
show, unsurprisingly, that pre-trained language models are extremely effective
in transfer learning. What is particularly interesting, though, is that
starting with random embeddings for tokens representing MWEs can lead to
comparative (and in some cases slightly better) scores. This suggests that
these tokens have at least a reasonable representation level, thus providing a
sample efficient method of learning embeddings for them. However, further
experiments on different tasks are required to test the extent to which these
tokens have been trained.

##  6 Conclusions and Future Work

In this work we presented a novel dataset of naturally occurring idiomatic MWE
usage in English and Portuguese, with associated tasks aimed at testing the
ability of language models to deal with idiomaticity. In addition, we ran a
number of experiments on these tasks.

In terms of idiomaticity detection, the results of our experiments show these
models achieve reasonable performance in the one-shot and few-shot settings,
but particularly struggle with the zero-shot setting, where the models
encounter unseen MWEs at inference time.

When it comes to the representation of idiomaticity, our experiments show that
while the use of the idiom principle does help in representing MWEs, these
gains do not transfer to a significant overall increase in performance on the
entire test split. The large number of MWEs makes including all of them in the
vocabulary impractical, likewise selectively training models with MWEs of
interest is impractical due to the cost of pre-training. This underscores the
need for a more nuanced approach to incorporating the idiom principle with
pre-trained language models. Additionally, in creating representations for
MWEs that are partially compositional, methods that make use of the
representations of constituent words such as attentive mimicking Schick and
Schütze (2019) might be beneficial and we intend to experiment with these
methods in future. We also find that pre-training is potentially an effective
way of learning these representations, although more experiments are required
to test these representations.

There are many avenues for future work using the data presented here,
including running cross-lingual experiments across different scenarios of data
availability. Although our experiments have been limited to the use of
transformer based pre-trained language models, the dataset and tasks we
present can be used with any language model. While this work provides a useful
dataset for the investigation of idiomaticity, we intend to expand this
dataset in order to cover a broader set of languages, and include a wider
range of idiomatic MWE types, including more syntactically flexible
expressions. One limitation of the dataset is that the paraphrases generated
are syntactically rigid, and for Task 2 the replacement sentences may not
always be grammatically correct (see Table 3). Although this is sufficient for
current purposes, future datasets could generate paraphrases per sentence
rather than per MWE.

### Acknowledgements

This work was partially supported by the UK EPSRC grant EP/T02450X/1 and the
CDT in Speech and Language Technologies and their Applications funded by UKRI
(grant number EP/S023062/1).

## References

  * Baldwin et al. (2003) Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. 2003.  An empirical model of multiword expression decomposability.  In _Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment_ , pages 89–96, Sapporo, Japan. Association for Computational Linguistics.
  * Baldwin and Villavicencio (2002) Timothy Baldwin and Aline Villavicencio. 2002.  Extracting the unextractable: A case study on verb-particles.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Biemann and Giesbrecht (2011) Chris Biemann and Eugenie Giesbrecht. 2011.  Distributional semantics and compositionality 2011: Shared task description and results.  In _Proceedings of the Workshop on Distributional Semantics and Compositionality_ , pages 21–28, Portland, Oregon, USA. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Constant et al. (2017) Mathieu Constant, Gülşen Eryiǧit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017.  Survey: Multiword expression processing: A Survey.  _Computational Linguistics_ , 43(4):837–892.
  * Cook et al. (2008) Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008.  The VNCTokens Dataset.  In _In proceedings of the MWE workshop. ACL_.
  * Cordeiro et al. (2019) Silvio Cordeiro, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019.  Unsupervised compositionality prediction of nominal compounds.  _Computational Linguistics_ , 45(1):1–57.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Farahmand et al. (2015) Meghdad Farahmand, Aaron Smith, and Joakim Nivre. 2015.  A multiword expression data set: Annotating non-compositionality and conventionalization for English noun compounds.  In _Proceedings of the 11th Workshop on Multiword Expressions_ , pages 29–33, Denver, Colorado. Association for Computational Linguistics.
  * Garcia et al. (2021) Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021.  Probing for idiomaticity in vector space models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3551–3564, Online. Association for Computational Linguistics.
  * Hashempour and Villavicencio (2020) Reyhaneh Hashempour and Aline Villavicencio. 2020.  Leveraging contextual embeddings and idiom principle for detecting idiomaticity in potentially idiomatic expressions.  In _Proceedings of the Workshop on the Cognitive Aspects of the Lexicon_ , pages 72–80, Online. Association for Computational Linguistics.
  * Hashimoto and Tsuruoka (2016) Kazuma Hashimoto and Yoshimasa Tsuruoka. 2016.  Adaptive joint learning of compositional and non-compositional phrase embeddings.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 205–215, Berlin, Germany. Association for Computational Linguistics.
  * Hendrickx et al. (2013) Iris Hendrickx, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2013.  SemEval-2013 task 4: Free paraphrases of noun compounds.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 138–143, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Kartsaklis et al. (2014) Dimitri Kartsaklis, Nal Kalchbrenner, and Mehrnoosh Sadrzadeh. 2014.  Resolving lexical ambiguity in tensor regression models of meaning.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 212–217, Baltimore, Maryland. Association for Computational Linguistics.
  * Katz and Giesbrecht (2006) Graham Katz and Eugenie Giesbrecht. 2006.  Automatic identification of non-compositional multi-word expressions using latent semantic analysis.  In _Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties_ , pages 12–19, Sydney, Australia. Association for Computational Linguistics.
  * Lin (1999) Dekang Lin. 1999.  Automatic identification of non-compositional phrases.  In _Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics_ , pages 317–324, College Park, Maryland, USA. Association for Computational Linguistics.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg S. Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’13, page 3111–3119, Red Hook, NY, USA. Curran Associates Inc.
  * Mitchell and Lapata (2010) Jeff Mitchell and Mirella Lapata. 2010.  Composition in distributional models of semantics.  _Cognitive science_ , 34(8):1388–1429.
  * Nandakumar et al. (2019) Navnita Nandakumar, Timothy Baldwin, and Bahar Salehi. 2019.  How well do embedding models capture non-compositionality? a view from multiword expressions.  In _Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP_ , pages 27–34, Minneapolis, USA. Association for Computational Linguistics.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Real et al. (2020) Livy Real, Erick Fonseca, and Hugo Goncalo Oliveira. 2020.  The assin 2 shared task: a quick overview.  In _International Conference on Computational Processing of the Portuguese Language_ , pages 406–412. Springer.
  * Reddy et al. (2011) Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011.  An empirical study on compositionality in compound nouns.  In _Proceedings of 5th International Joint Conference on Natural Language Processing_ , pages 210–218, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
  * Reimers et al. (2016) Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016.  Task-oriented intrinsic evaluation of semantic textual similarity.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 87–96, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Sag et al. (2002) Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann A. Copestake, and Dan Flickinger. 2002.  Multiword expressions: A pain in the neck for nlp.  In _Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing_ , CICLing ’02, page 1–15, Berlin, Heidelberg. Springer-Verlag.
  * Savary et al. (2017) Agata Savary, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and Antoine Doucet. 2017.  The PARSEME shared task on automatic identification of verbal multiword expressions.  In _Proceedings of the 13th Workshop on Multiword Expressions (MWE 2017)_ , pages 31–47, Valencia, Spain. Association for Computational Linguistics.
  * Schick and Schütze (2019) Timo Schick and Hinrich Schütze. 2019.  Attentive mimicking: Better word embeddings by attending to informative contexts.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 489–494, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Schneider et al. (2016) Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Meredith Green, Abhijit Suresh, Kathryn Conger, Tim O’Gorman, and Martha Palmer. 2016.  A corpus of preposition supersenses.  In _Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)_ , pages 99–109, Berlin, Germany. Association for Computational Linguistics.
  * Schneider et al. (2014) Nathan Schneider, Spencer Onuffer, Nora Kazour, Emily Danchik, Michael T. Mordowanec, Henrietta Conrad, and Noah A. Smith. 2014.  Comprehensive annotation of multiword expressions in a social web corpus.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 455–461, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Schneider and Smith (2015) Nathan Schneider and Noah A. Smith. 2015.  A corpus and model integrating multiword expressions and supersenses.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1537–1547, Denver, Colorado. Association for Computational Linguistics.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 815–823.
  * Schulte im Walde et al. (2016) Sabine Schulte im Walde, Anna Hätty, Stefan Bott, and Nana Khvtisavrishvili. 2016.  GhoSt-NN: A representative gold standard of German noun-noun compounds.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 2285–2292, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Shwartz and Dagan (2019) Vered Shwartz and Ido Dagan. 2019.  Still a pain in the neck: Evaluating text representations on lexical composition.  _Transactions of the Association for Computational Linguistics_ , 7:403–419.
  * Sinclair et al. (1991) J. Sinclair, L. Sinclair, and R. Carter. 1991.  _Corpus, Concordance, Collocation_.  Describing English language. Oxford University Press.
  * Tu and Roth (2012) Yuancheng Tu and Dan Roth. 2012.  Sorting out the most confusing English phrasal verbs.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 65–69, Montréal, Canada. Association for Computational Linguistics.
  * Venkatapathy and Joshi (2005) Sriram Venkatapathy and Aravind Joshi. 2005.  Measuring the relative compositionality of verb-noun (V-n) collocations by integrating features.  In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_ , pages 899–906, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Yu and Ettinger (2020) Lang Yu and Allyson Ettinger. 2020.  Assessing phrasal representation and composition in transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4896–4907, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Dataset Statistics

Detailed statistics for the English and Portuguese datasets are shown in Table
10 and Table 11, respectively. The train, dev and set breakdowns are shown in
the leftmost column, with the further breakdown of the train set into zero-
shot, one-shot and few-shot setups. Note here that the one-shot data is
contained within the few-shot data. The MWEs column is the number of MWEs that
the examples span - the one-shot and few-shot setups contain all the MWEs from
the dev and test sets, but the examples are different. The next columns show
the fine-grained and coarse-grained breakdown of the dataset, used in Task 1
Subtask B and Task 1 Subtask A, respectively.

Set |  |  | Non-Idiomatic (1) | Idiomatic (0) |
---|---|---|---|---|---
|  | MWEs | Lit | PN | Tot | 1 | 2 | 3 | Meta | Tot | Tot
train | zero-shot | 163 | 1110 | 455 | 1565 | 1614 | 92 | 8 | 48 | 1762 | 3327
(one-shot) | 60 | 29 | 26 | 55 | 25 | 5 | 0 | 2 | 32 | 87
few-shot | 60 | 135 | 50 | 185 | 81 | 11 | 0 | 5 | 97 | 282
total | 223 | 1245 | 505 | 1750 | 1695 | 103 | 8 | 53 | 1859 | 3609
dev |  | 30 | 174 | 110 | 284 | 157 | 14 | 0 | 11 | 182 | 466
test |  | 30 | 271 | 63 | 334 | 118 | 24 | 0 | 7 | 149 | 483
total |  | 223 | 1690 | 678 | 2368 | 1970 | 141 | 8 | 71 | 2190 | 4558
Table 10: Breakdown of the English dataset. Set |  |  | Non-Idiomatic (1) | Idiomatic (0) |
---|---|---|---|---|---
|  | MWEs | Lit | PN | Tot | 1 | 2 | 3 | Meta | Tot | Tot
train | zero-shot | 73 | 284 | 107 | 391 | 697 | 55 | 2 | 19 | 773 | 1164
(one-shot) | 40 | 17 | 8 | 25 | 26 | 2 | 0 | 0 | 28 | 53
few-shot | 40 | 55 | 14 | 69 | 80 | 6 | 0 | 1 | 87 | 156
total | 113 | 339 | 121 | 460 | 777 | 61 | 2 | 20 | 860 | 1320
dev |  | 20 | 96 | 23 | 119 | 137 | 16 | 0 | 1 | 154 | 273
test |  | 20 | 94 | 20 | 114 | 151 | 9 | 0 | 5 | 165 | 279
total |  | 113 | 529 | 164 | 693 | 1065 | 86 | 2 | 26 | 1179 | 1872
Table 11: Breakdown of the Portuguese dataset.

##  Appendix B Experimental Procedure

###  B.1 Task 1 Subtask A

The Task 1 experiments were run on NVIDIA Tesla K80s. For the first subtask,
we ran a range of experiments, varying the model used, whether context was
used, and whether the MWE was used. We ran each experiment for 9 epochs with
five seeds (0 - 5). For all experiments, we used a max sequence length of 128
and a learning rate of 2e-5. The standard tokenizers for each model were used
for tokenizing the input. The results for the best performing seed and epoch
(by F1 score) for each experiment are shown in Table 12, with approximate
training run times (for one seed for nine epochs). We started with the zero-
shot experiments, then took the best performing models, and continued training
them from the best epoch for another 9 epochs in the one-shot and few-shot
setups.

###  B.2 Task 1 Subtask B

For the second subtask, we took the best-performing experimental settings from
Subtask A: XLNET base (cased) for English and BERT base multilingual (cased)
for Portuguese, excluding context but not including the MWE since we instead
pass the relevant paraphrase of the MWE (either correct or incorrect). These
models were then trained on the multiclass data, again for 9 epochs and with
five seeds (0-5). Again, the best-performing models in the zero-shot setup
were continued training from the best epoch for another 9 epochs in the few-
shot and one-shot setups. These experiments are shown in Table 13. Training
times are increased due to the larger dataset from generation of negative
samples.

| Problem Setup | Model | Context? | MWE? | Train Time | Dev Accuracy | Dev F1
---|---|---|---|---|---|---|---
English | zero-shot | BERT base (cased) | No | No | ~1 hour | 0.732 | 0.724
BERT base (cased) | Yes | No | ~1 hour | 0.732 | 0.717
BERT base (cased) | Yes | Yes | ~1 hour | 0.785 | 0.779
BERT base (cased) | No | Yes | ~1 hour | 0.796 | 0.785
| BERT base (uncased) | No | Yes | ~1 hour | 0.777 | 0.77
| XLNet base (cased) | No | Yes | ~1 hour | 0.828 | 0.823
|  | DistilBERT base (cased) | No | Yes | ~1 hour | 0.768 | 0.757
|  | RoBERTa base (cased) | No | Yes | ~1 hour | 0.807 | 0.801
| one-shot | XLNet base (cased) | No | Yes | +~5 mins | 0.903 | 0.897
| one-shot | XLNet base (cased) | Yes | No | +~5 mins | 0.719 | 0.689
| one-shot | XLNet base (cased) | No | No | +~5 mins | 0.775 | 0.755
| few-shot | XLNet base (cased) | No | Yes | +~1min | 0.961 | 0.959
| few-shot | XLNet base (cased) | Yes | No | +~1min | 0.807 | 0.782
| few-shot | XLNet base (cased) | No | No | +~1min | 0.813 | 0.792
Portuguese | zero-shot | XLM-RoBERTa base (cased) | No | No | ~1 hour | 0.604 | 0.593
XLM-RoBERTa base (cased) | Yes | No | ~1 hour | 0.56 | 0.542
XLM-RoBERTa base (cased) | Yes | Yes | ~1 hour | 0.714 | 0.696
XLM-RoBERTa base (cased) | No | Yes | ~1 hour | 0.729 | 0.703
| BERT base multilingual (cased) | No | Yes | ~1 hour | 0.707 | 0.686
one-shot | XLM-RoBERTa base (cased) | No | Yes | +~5 mins | 0.879 | 0.877
| one-shot | XLM-RoBERTa base (cased) | Yes | No | +~5 mins | 0.615 | 0.605
| one-shot | XLM-RoBERTa base (cased) | No | No | +~5 mins | 0.641 | 0.638
| few-shot | XLM-RoBERTa base (cased) | No | Yes | +~1min | 0.927 | 0.926
| few-shot | XLM-RoBERTa base (cased) | Yes | No | +~1min | 0.656 | 0.655
| few-shot | XLM-RoBERTa base (cased) | No | No | +~1min | 0.799 | 0.796
Table 12: Dev set results for Task 1 Subtask A  | Problem Setup | Model | Train Time | Dev Accuracy | Dev F1
---|---|---|---|---|---
En | zero-shot | XLNet base (cased) | ~2.5 hours | 0.883 | 0.852
one-shot | XLNet base (cased) | +~20 mins | 0.938 | 0.923
few-shot | XLNet base (cased) | +~1 hour | 0.947 | 0.933
Pt | zero-shot | XLM-RoBERTa base (cased) | ~1 hour | 0.886 | 0.843
one-shot | XLM-RoBERTa base (cased) | +~5 mins | 0.888 | 0.852
few-shot | XLM-RoBERTa base (cased) | +~20 mins | 0.931 | 0.909
Table 13: Dev set results for Task 1 Subtask B

###  B.3 Task 2

Pre-training models was done using NVIDIA Tesla V100s and took approximately
15 hours for each of the two models in English (BERT base on “all replaced”
and “select replaced”) and 5 hours for each model in Portuguese (BERT base
multilingual). Due to time and resource limitations, we pre-train models only
once. All models were pre-trained for 5 epochs based on the evaluation on a
development set and our initial experiments which showed that further pre-
training did not improve results.

For Subtask A, fine-tuning these models using the Sentence BERT architecture
(so as to be able to compare the resultant embeddings using cosine similarity)
was done using NVIDIA K80 GPUs for English and took approximately 6 minutes
per seed. Since we tested four variations (original BERT, BERT tokenized but
not pre-trained, BERT all tokenized and select tokenized) each with five
seeds, these experiments took a total of about two hours. The multilingual
models required the use of NVIDIA Tesla V100s due to their larger size and
took about 3 minutes to train each model (per seed) and consequently took a
total of about an hour to train. The best model was picked based on the
performance on the STS dataset they were trained on (i.e. the STS benchmark
dataset for English and ASSIN2 for Portuguese).

Subtask B similarly required the use of NVIDIA K80 GPUs for English and NVIDIA
Tesla V100s for Portuguese. We select the best models fine-tuned using the
Sentence BERT architecture (with no pre-training) from Subtask A and continue
pre-training with MWE specific data. This process took approximately 6 minutes
per model in English and 3 minutes in Portuguese leading to a total of about
30 minutes and 15 minutes respectively.

All fine-tuning was done for as many epochs as was required to see a drop in
performance on the corresponding development set.

###  B.4 Larger Models

Exploratory experiments on Task 1 showed that the larger language models
performed worse than the base ones, and thus these were the ones we used in
our experiments.

For task 2, we use the smaller base models due to the limited amount of pre-
train data, which we believe would make the use of larger models impractical.


_Journal of Fractional Calculus and Applications_ ,
Vol. 3(S). July, 11, 2012 (Proc. of the 4th. Symb. of Fractional Calculus and
Applications)
No. 6, PP. 1 – 8
ISSN: 2090-5858.
http://www.fcaj.webs.com/

#  A New Technique for the Calculation of Effective Mesonic Potential at
Finite Temperature in the Logarithmic Quark-Sigma Model

M. Abu-Shady  M. Abu-shady
Faculty of Science, Menofia University, Menofia, Egypt
<EMAIL_ADDRESS>

###### Abstract.

The logarithmic sigma model describes the interactions between quarks via
sigma and pion exchanges. The effective mesonic potential is extended to the
finite temperature and it is numerically calculated using n-midpoint rule.
Meson properties such as the phase transition, the sigma and pion masses, and
the critical point temperature are examined as functions of temperature. The
obtained results are compared with other approaches. We conclude that the
calculated effective potential is successfully to predict the meson
properties.

###### Key words and phrases:

Finite-temperature field theory, Chiral Lagrangians, Midpoint

###### 1991 Mathematics Subject Classification:

81V05, 81V25, 82B26, 82B80

Proc. of the 4th. Symb. of Frac. Calcu. Appl. Faculty of Science Alexandria
University, Alexandria, Egypt July, 11, 2012

##  1\. Introduction

The study of matter at very high temperature and densities is of interest
because of its relevance to particle physics and astrophysics. According to
the standard big bang model, it is believed that a series of phase transitions
happened at the early stages of the evolution of universe. The QCD phase
transition being one of them. The lattice QCD and effective field theories are
two main approaches to calculate the phase transition and meson properties at
finite temperature [1]. This subject has been under intense theoretical study
using various effective field theory models, such as the Namu-Jona-Lasinio
Model [2-4], a linear sigma model [5-8]. One of the effective models in
describing baryon properties is the linear sigma model, which was suggested
earlier by Gell-Mann and Levy [9] to describe the nucleons interacting via
sigma $(\sigma)$ and pion $(\mathbf{\pi})$ exchanges. At finite temperature,
the model gives a good description of the phase transition by using the
Hartree approximation [1, 10-12] within the Cornwall–Jackiw–Tomboulis (CJT)
formalism [13]. However, there exists serious difficulty concerning the
renormalization of the CJT effective action in the Hartree approximation.
Baacke and Michalski [14] indicated that the phase transition can be obtained
beyond the large $N$ and Hartree approximation through the systematic
expansions are bases on the resummation scheme by Cornwall-Jackiw-Tombonlis
and 2PI scheme.

The aim of this work is to calculate the effective logarithmic mesonic
potential, the phase transition, and meson masses at finite temperature. The
logarithmic mesonic potential at zero temperature was suggested in Ref. [15]
to provide a good description of hadron properties. The used method is
different as a new technique in comparison with other works as in Refs. [5-8].
In addition, the pressure is investigated as a function temperature.

This paper is organized as follows: In Sec. 2, the linear sigma model at zero
temperature and finite temperature are explained briefly. The numerical
calculations and discussion of the results are presented in Sec. 3. Summary
and conclusion is presented in Sec. 4.

##  2\. The Logarithmic Quark-Sigma Model

###  2.1. The Logarithmic Potential at zero Temperature

The Lagrangian density of quark sigma model that describes the interactions
between quarks via the $\sigma-$and $\mathbf{\pi}-$meson exchange [15]. The
Lagrangian density is,

$L\left(r\right)=i\overline{\Psi}\partial_{\mu}\gamma^{\mu}\Psi+\frac{1}{2}\left(\partial_{\mu}\sigma\partial^{\mu}\sigma+\partial_{\mu}\mathbf{\pi}.\partial^{\mu}\mathbf{\pi}\right)+g\overline{\Psi}\left(\sigma+i\gamma_{5}\mathbf{\tau}.\mathbf{\pi}\right)\Psi-U^{T(0)}(\sigma,\mathbf{\pi)},$
(1)

with

$U^{T(0)}\left(\sigma,\mathbf{\pi}\right)=\lambda_{1}^{2}(\sigma^{2}+\mathbf{\pi}^{2})-\lambda_{2}^{2}\log(\sigma^{2}+\mathbf{\pi}^{2})+m_{\pi}^{2}f_{\pi}\sigma,$
(2)

$U^{T(0)}\left(\sigma,\mathbf{\pi}\right)$ is the meson-meson interaction
potential where $\Psi,\sigma$ and $\mathbf{\pi}$ are the quark, sigma, and
pion fields, respectively. In the mean-field approximation the meson fields
are treated as time-independent classical fields. This means that we replace
power and products of the meson fields by corresponding powers and products of
their expectation values. The meson-meson interactions in Eq. (2) lead to
hidden chiral $SU(2)\times SU(2)$ symmetry with $\sigma\left(r\right)$ taking
on a vacuum expectation value

$\ \ \ \ \ \ \left\langle\sigma\right\rangle=-f_{\pi},$ (3)

where $f_{\pi}=93$ MeV is the pion decay constant. The final term in Eq. (2)
is included to break the chiral symmetry. It leads to partial conservation of
axial-vector isospin current (PCAC). The parameters $\lambda^{2},\nu^{2}$ can
be expressed in terms of $\ f_{\pi}$, the masses of mesons as,

$\lambda_{1}^{2}=\frac{1}{4}(m_{\sigma}^{2}+m_{\pi}^{2}),$ (4)
$\lambda_{2}^{2}=\frac{f_{\pi}^{2}}{4}(m_{\sigma}^{2}-m_{\pi}^{2}).$ (5)

###  2.2. The Effective Mesonic Potential at Finite-Temperature

In Eq. (2), The effective potential is extended to calculate the chiral
interacting mesons with quarks at finite temperature. The quarks are
considered a heat bath in local thermal equilibrium

$U_{eff}(\sigma,\mathbf{\pi,}T\mathbf{)}=U^{T(0)}(\sigma,\mathbf{\pi)-}24T\mathop{\displaystyle\int}\frac{d^{3}p}{\left(2\pi\right)^{3}}\ln(1+e^{\frac{-\sqrt{p^{2}+g^{2}(\sigma^{2}+\mathbf{\pi}^{2})}}{T}}),$
(6)

the first term is the potential in the tree level defines in Eq. (2) and the
second term is for the chiral meson fields interacts with quarks at finite
temperature and zero-chemical potential. Non-zero values of the chiral fields
in the chiral broken phase dynamically generate a quark mass $m_{q}=gf_{\pi}$.
The integration is taken over momentum volume (for details, see Ref. [16]).

##  3\. Numerical Calculations and Discussion of Results

###  3.1. Numerical Calculations

The purpose of this section is to calculate the effective mesonic potential
$\sigma$ and $\mathbf{\pi}$\- masses, and the pressure. We rewrite Eq. (6) as
follows

$U_{eff}(\sigma,\mathbf{\pi,}T\mathbf{)}=U^{T(0)}(\sigma,\mathbf{\pi)-}\frac{12T}{\pi^{2}}\int_{0}^{\infty}p^{2}dp(\ln(1+e^{\frac{-\sqrt{p^{2}+g^{2}(\sigma^{2}+\mathbf{\pi}^{2})}}{T}}).$
(7)

Eq. (7) is written in the dimensionless form as follows

$U_{eff}(\sigma,\mathbf{\pi,}T\mathbf{)}=f_{\pi}^{4}[U^{T(0)}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{)-}\frac{12T^{\prime}}{\pi^{2}}\int_{0}^{\infty}p^{{}^{\prime}2}(\ln
e^{{}^{\left(-\frac{1}{T^{\prime}}\sqrt{P^{\prime 2}+g^{2}(\sigma^{\prime
2}+\chi^{\prime 2})}\right)}}+1)dp^{\prime}],$ (8)

where

$U^{T(0)}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{)=}\lambda_{1}^{{}^{\prime}2}(\sigma^{\prime
2}+\mathbf{\pi}^{{}^{\prime}2})-\lambda_{2}^{{}^{\prime}2}\log
f_{\pi}^{2}(\sigma^{{}^{\prime}2}+\mathbf{\pi}^{\prime
2})+m_{\pi}^{{}^{\prime}2}\sigma^{\prime},$ (9)

and

$\lambda_{1}^{{}^{\prime}2}=\frac{1}{4}(m_{\sigma}^{{}^{\prime}2}+m_{\pi}^{{}^{\prime}2}),\lambda_{2}^{\prime
2}=\frac{1}{4}(m_{\sigma}^{\prime 2}-m_{\pi}^{{}^{\prime}2}),$ (10)

where
$\sigma^{\prime},\mathbf{\pi}^{\prime},T^{\prime},m_{\pi}^{{}^{\prime}},$and
$m_{\sigma}^{{}^{\prime}}$ are in unit of $f_{\pi}.$ Therefore
$U^{T(0)}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{)}$ is the
dimensionless form of $U^{T(0)}(\sigma,\mathbf{\pi)}$. Substituting
$p^{\prime}=-\ln y$ into Eq. (8), we obtain:

$\displaystyle U_{eff}(\sigma,\mathbf{\pi,}T\mathbf{)}$ $\displaystyle=$
$\displaystyle
f_{\pi}^{4}[U^{T(0)}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{)-}$
$\displaystyle-\frac{12T^{\prime}}{\pi^{2}}\int_{0}^{1}\frac{(\ln
y)^{2}}{y}\ln\left(\exp\left(-\frac{1}{T^{\prime}}\sqrt{(\ln
y)^{2}+g^{2}(\sigma^{\prime 2}+\mathbf{\pi}^{\prime
2})}\right)+1\right)dy\allowbreak],\ \ \ \ \ $

hence we can write the dimensionless form of
$U_{eff}(\sigma,\mathbf{\pi,}T\mathbf{)}$ as follows:

By using midpoint rule, we obtain the approximate integral as follows:

$U_{eff}(\sigma,\mathbf{\pi,}T\mathbf{)}=f_{\pi}^{4}[U^{T(0)}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{)-}\frac{12T^{\prime}}{\pi^{2}}A\ln\left(\exp\left(-\frac{1}{T^{\prime}}\sqrt{g^{2}\left(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2}\right)+B}\right)+1\right)],$ (12)

where

$A=\frac{1}{n}\sum_{i=0}^{n}\frac{1}{\frac{1}{n}i+\frac{1}{2n}}\ln^{2}\left(\frac{1}{n}i+\frac{1}{2n}\right)\allowbreak,\
\ \ B=\sum_{i=0}^{n}\ln^{2}\left(\frac{1}{n}i+\frac{1}{2n}\right).$ (13)

(for details, see Refs. [17, 18]). In Ref. [19], the authors applied the
second derivation of the effective potential respect to $\sigma^{\prime}$ and
$\pi^{\prime}$ to obtain the effective meson masses. Then, the first
derivative of the effective potential
$U_{eff}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T^{\prime}\mathbf{)}$
is given by

$\frac{\partial
U_{eff}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T\mathbf{)}}{\partial\sigma}=\frac{1}{f_{\pi}}\frac{\partial
U_{eff}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T\mathbf{)}}{\partial\sigma^{\prime}}=f_{\pi}^{3}[\frac{\partial
U_{0}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T\mathbf{)}}{\partial\sigma^{\prime}}\mathbf{-}\frac{12T^{\prime}}{\pi^{2}}\frac{d_{1}d_{2}}{d_{3}+d_{3}d_{2}}],$
(14)

where

$\displaystyle d_{1}$ $\displaystyle=$ $\displaystyle-Ag^{2}\sigma^{\prime},\
\ d_{2}=\exp\left(-\frac{1}{T^{\prime}}\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2})}\right),\ \ $ $\displaystyle d_{3}\ $
$\displaystyle=$ $\displaystyle T^{\prime}\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2})}.\TCItag{15}$ (4)

Then, we obtain the effective sigma mass as follows

$m_{\sigma}^{2}(T)=\frac{\partial^{2}U_{eff}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T\mathbf{)}}{f_{\pi}^{2}\partial\sigma^{\prime
2}}=f_{\pi}^{2}[\frac{\partial^{2}U_{0}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T\mathbf{)}}{\partial\sigma^{\prime
2}}\mathbf{-}\frac{12T}{\pi^{2}}^{\prime}\frac{\partial}{\partial\sigma^{\prime}}(\frac{d_{1}d_{2}}{d_{3}+d_{3}d_{2}})],$
(16)
$m_{\sigma}(T)=f_{\pi}[\frac{\partial^{2}U_{0}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,}T\mathbf{)}}{\partial\sigma^{\prime
2}}\mathbf{-}\frac{12T^{\prime}}{\pi^{2}}\frac{\partial}{\partial\sigma^{\prime}}(\frac{d_{1}d_{2}}{d_{3}+d_{3}d_{2}})]^{\frac{1}{2}},$
(17)

where

$\displaystyle\frac{\partial}{\partial\sigma^{\prime}}(\frac{d_{1}d_{2}}{d_{3}+d_{3}d_{2}})$
$\displaystyle=$ $\displaystyle
d_{1}\frac{d_{2}^{\prime}}{d_{3}+d_{2}d_{3}}+d_{2}\frac{d_{1}^{\prime}}{d_{3}+d_{2}d_{3}}-$
(5) $\displaystyle
d_{1}\frac{d_{2}}{\left(d_{3}+d_{2}d_{3}\right)^{2}}\allowbreak\left(d_{3}^{\prime}+d_{2}d_{3}^{\prime}+d_{3}d_{2}^{\prime}\right),\TCItag{18}$

with

$d_{1}^{\prime}=\frac{\partial d_{1}}{\partial\sigma^{\prime}}=\allowbreak-
Ag^{2},$ (19)

$d_{2}^{\prime}=\frac{\partial
d_{2}}{\partial\sigma^{\prime}}=\allowbreak-\frac{1}{T^{\prime}}g^{2}\frac{\sigma^{\prime}}{\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime
2})}}\exp\left(-\frac{1}{T}\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2})}\right)\allowbreak,$ (20)
$d_{3}^{\prime}=\frac{\partial d_{3}}{\partial\sigma^{\prime}}=\allowbreak
Tg^{2}\frac{\sigma^{\prime}}{\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2})}}.$ (21)

Similarly, we obtain the effective pion mass as follows

$m_{\pi}(T)=f_{\pi}[\frac{\partial^{2}U(\sigma^{\prime},\mathbf{\pi}^{\prime})}{\partial\mathbf{\pi}^{\prime
2}}-.\frac{12}{\pi^{2}}T^{\prime}\frac{\partial}{\partial\mathbf{\pi}^{\prime}}(\frac{d_{4}d_{2}}{d_{3}+d_{3}d_{2}})]^{\frac{1}{2}},$
(22)

where

$d_{4}=-Ag^{2}\mathbf{\pi}^{\prime},$ (23)

$\displaystyle\frac{\partial}{\partial\mathbf{\pi}^{\prime}}(\frac{d_{4}d_{2}}{d_{3}+d_{3}d_{2}})$
$\displaystyle=$ $\displaystyle
d_{2}\frac{d_{4}^{\prime}}{d_{3}+d_{2}d_{3}}+d_{4}\frac{d_{2}^{\prime}}{d_{3}+d_{2}d_{3}}-$
(6) $\displaystyle
d_{2}\frac{d_{4}}{\left(d_{3}+d_{2}d_{3}\right)^{2}}\allowbreak\left(d_{3}^{\prime}+d_{2}d_{3}^{\prime}+d_{3}d_{2}^{\prime}\right),\TCItag{24}$

where

$d_{4}^{\prime}=\frac{\partial
d_{4}}{\partial\mathbf{\pi}^{\prime}}=\allowbreak-Ag^{2},$ (25)

$d_{2}^{\prime}=\frac{\partial
d_{2}}{\partial\mathbf{\pi}^{\prime}}=-\frac{1}{T}g^{2}\frac{\mathbf{\pi}^{\prime}}{\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime
2})}}\exp\left(-\frac{1}{T}\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2})}\right),$ (26)

$d_{3}^{\prime}=\frac{\partial
d_{3}}{\partial\mathbf{\pi}^{\prime}}=\allowbreak
Tg^{2}\frac{\mathbf{\pi}^{\prime}}{\sqrt{B+g^{2}(\sigma^{\prime
2}+\mathbf{\pi}^{\prime 2})}}.$ (27)

In Ref. [16], the pressure is given in the dimensionless form as follow:

$P^{\prime}(\sigma^{\prime},\mathbf{\pi}^{\prime},T)=U^{T(0)}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{)}-U_{eff}(\sigma^{\prime},\mathbf{\pi}^{\prime}\mathbf{,T}^{\prime}\mathbf{),}$
(28)

##  4\. Results and Discussion

In Eq. (7), The integration is solved by the n-midpoint algorithm and the
index n is taken $n=1000$ to get a good accuracy for a numerically
integration. In Ref. [20], the authors used a different method for calculating
the integration in the effective potential and they obtained it as a series of
$M^{2}=m^{2}+\frac{\lambda}{2}\phi^{2}$ , where the $m,$ $\lambda$ are
parameters of the model. The difficulty in this potential is to determine a
critical temperature $T_{c}.$ The two terms were taken only in the expression
of the potential since the increase of terms more than two terms will be the
critical point is a complex value, leading $T_{c}$ is not a physics quantity.
In Refs. [1-4], authors used double bubble graphs instead of summing infinite
set of daisy and superdaisy graphs using the tree level propagators.
Therefore, the Feynman diagrams are needed. In the present work, The
n-midpiont rule is used to avoid the difficulties in the above approaches.
Hence, we did not need to apply Feynman diagrams. The parameters of the model
such as $m_{\pi}=140$ MeV, $m_{\sigma}=600$ MeV$,~{}f_{\pi}=93$ MeV, and
coupling constant $g$ at zero-temperature are used as the initial parameters
at the finite temperature. In addition, the effective pion and sigma masses
are obtained as a second derivative respect to meson field. This method is
used extensively in other works such as in the Ref. [19]. In this section, we
examine the meson properties and the phase transition which depend on the
calculation of the effective mesonic potential at finite temperature. We
replaced the normal potential in the linear sigma model [5] at zero
temperature by logarithmic mesonic potential [15]. In our previous work [15].
The logarithmic potential was successfully to predict hadron properties at
zero temperature.

In Figs. (1, 2, 3), we investigate the behavior of the sigma and pion masses
as functions of temperature. Moreover, the effect of the sigma mass and the
coupling constant $g$ on a critical point temperature in the presence of the
explicit symmetry breaking ($m_{\pi}\neq 0)$ and the chiral limit(
$m_{\pi}=0$) is investigated. In Fig. 1, the sigma and pion masses are plotted
as functions of temperature at the presence of the explicit symmetry breaking
term. The sigma mass decreases with increasing temperature and the pion mass
increases with increasing temperature. The two curves crossed at a critical
point temperature $T_{c}$ where the sigma and pion masses have the same
massive value. At $m_{\sigma}=600$ MeV, we find the critical point temperature
$T_{c}$ =233 MeV. By increasing the sigma mass up to $m_{\sigma}=700$ MeV
leads to $T_{c}$ = 309 MeV. In comparison with lattice QCD results [19], the
critical point temperature is found in the range $T_{c}=100\ $to 300 MeV.
Therefore the present result is in agreement with lattice QCD results. Nemoto
et al. [19] calculated the critical point temperature equal 230 MeV using the
original sigma model. Abu-shady [5] calculated the critical point temperature
equal 226 MeV in the original sigma model. Therefore, the present result is in
good agreement with Refs. [5, 19]. In Fig. 2, the sigma and pion masses are
plotted as functions of temperature in the chiral limit ( $m_{\pi}=0$). A
similar behavior is obtained as in the case of the explicit symmetry term
expect the pion mass equal zero at zero temperature. In comparison with
original sigma model as in Refs. [1, 5, 19], we found that the behavior of the
sigma and pion masses are in good agreement with Refs. [1, 5, 19]. In Fig. 3,
we examine the effect of coupling constant $g$ on the critical point
temperature. We find that an increase on the coupling constant $g$ leads to
decrease in the critical point temperature g = 3.76 to g = 4.48 corresponding
to $T_{c}=$ 243 MeV to $T_{c}=191$ MeV, respectively. In Fig. (4), the
effective potential is plotted as a function of temperature. We note that the
potential decreases with increasing temperature. In order to get more insight
into the nature of the phase transition and verify that the order phase
transition is a second-order phase transition. We calculate the effective
potential $U_{eff}(\Phi\mathbf{,}T\mathbf{)}$ as a function of phase
transition ($\Phi=\sqrt{\sigma^{2}+\mathbf{\pi}^{2}})$ as seen in Fig. (4).
The shape of the potential confirms that the phase transition is the second-
order. Since it exhibits can degenerate one minima at $\Phi\neq 0$. The
indication of the second-order phase transition has been reported in many
works [14, 21- 23]. Also, We note the strong increase in the temperature T
which unchanged the shape of the potential

Next, we need to examine the effect of finite temperature on the behavior of
pressure where the quarks takes as a heat bath. In Fig. (5), the pressure is
plotted as a function of temperature. The pressure increases with increasing
temperature. Also, we note that the pressure value at lower-values of
temperature is not sensitive in comparison with the pressure value at the
higher-values of temperature. Hence the effect of largest values of
temperature is more affected on the value of pressure. Berger and Christov
[24] found that the pressure increases with increasing temperature in hot
medium using the NJL model in the mean field approximation. The present
behavior is in agreement with Ref. [24].

##  5\.  Summary and Conclusion

In this work, the effective mesonic potential is calculated by using
n-midpoint algorithm in the logarithmic quark model. The meson properties, the
phase transition, and the pressure are calculated using the effective mesonic
potential. We summarized the following points:The behavior of sigma and pion
masses as functions of temperature are investigated. A comparison with
original sigma model is presented. The increase of sigma mass
($m_{\sigma})~{}$leads to increase the critical point temperature. The
increase of the coupling constant ($g)$ leads to decrease the critical point
temperature. The phase transition is predicted as a second-order phase
transition which agrees with other works. The critical point temperature and
the pressure are calculated and are in agreement with other works. Therefore,
the calculated effective logarithmic potential is successful to predict the
meson masses, the phase transition, and the pressure at finite temperature.

## References

  * [1] N. Petropoulos, Linear sigma model and chiral symmetry at finite temperature, J. Phys. G 25, 2225 (1999).
  * [2] M. Asakawa and K. Yazaki, Chiral restoration at finite density and temperature, Nucl. Phys. A 504, 668 (1989).
  * [3] J. Berges and K. Rajagopal, Color superconductivity and chiral symmetry restoration at non-zero baryon density and temperature, Nucl. Phys. B 538, 215 (1999).
  * [4] O. Scavenius, A. Mocsy, I. N. Mishustin, and D. H. Rischke, Chiral phase transition within effective models with constituent, Phys. ReV. C 64, 045202 (2001).
  * [5] M. Abu-Shady, Meson properties at finite temperature in the linear sigma model, Int. J. Theor. Phys. 49, 2425, (2010).
  * [6] M. Abu-Shady and H. Mansour, Quantized linear sigma model at finite temperature and nucleon properties, Phys. ReV. C 85, 055204 (2012).
  * [7] M. Abu-Shady and H. Mansour, The linear sigma model with higher-order mesonic interactions at finite Temperature, (In preparation).
  * [8] M. Abu-Shady, Int. J. Mod. Phys. E, The Effect of finite temperature on the nucleon properties in the extended linear sigma model, (2012) (accepted).
  * [9] M. Gell-Mann and M. Levy, The Axial Vector Current in Beta Decay, Nuovo Cim. 16, 705 (1960).
  * [10] G. Amelino-Camelia and S. Y. Pi, Self-consistent improvement of the finite-temperature effective potential, Phys. Rev. D 47, 2356 (1993).
  * [11] G. Amelino-Camelia, Thermal effective potential of the O(N) linear sigma model, Phys. Lett. B 407, 268 (1997).
  * [12] J. T. Lenaghan and D. H. Rischke, The O(N) model at finite temperature, J. Phys. G 26, 431 (2000).
  * [13] J. M. Cornwall, R. Jackiw and E. Tomboulis, Effective action for composite operators, Phys. ReV. D 10, 2428 (1974).
  * [14] J. Baacke and S. Michalski, The O(N) linear sigma model at finite temperature beyond the Hartree approximation, Phys. ReV. D 67, 085006 (2003).
  * [15] M. Abu-shady and M. Rashdan, Effect of logarithmic mesonic potential on nucleon properties in the coherent-pair approximation, Phys. ReV. C 81, 015203 (2010).
  * [16] M. Nahrgang and M. Bleiher, Fluid dynamics with a critical point, Acta Phys. Polonic B 2, 405 (2009)
  * [17] D. Jogerman, Math. Computation, Investigation of a Modified Mid-Point Quadrature Formula, 20, 79 (1966).
  * [18] R. Mountain, Generalizations of midpoint rules, J. Mathematics. 1, 603 (1971).
  * [19] Y. Nemoto, K. Naito and M. Oka, Effective potential of the O(N) linear sigma model at finite temperature, Eur. Phys. J. A 9, 245 (2000).
  * [20] L. Dolan and R. Jackiw, Symmetry behavior at finite temperature, Phys. ReV. D 9, 12, (1974).
  * [21] K. Rajagopal and F. Wilkzek, Static and dynamic critical phenomena at a second order QCD phase transition, Nucl. Phys. B399, 395 (1993).
  * [22] H. Nakkagaw and H. Yokota, Rg Improvement of the Effective Potential at Finite Temperature, Mod. Phys. Lett. A 11, 2259 (1996).
  * [23] K. Ogure and J. Sato, Critical properties of the O(N) invariant scalar model using the auxiliary-mass method at finite temperature, Phys. ReV. D 58, 085010 (1998).
  * [24] J. Berger and C. Christov, Phase transition and nucleon as a soliton in the Nambu-Jona-Lasinio model, Nucl. Phys. A 609, 537 (1996).