# TaxoLLaMA: WordNet-based Model
for Solving Multiple Lexical Sematic Tasks

Viktor Moskvoretskii1,2, Ekaterina Neminova1, Alina Lobanova1,
Alexander Panchenko2,3, and Irina Nikishina4
1HSE University, 2Skoltech, 3AIRI, 4Universität Hamburg
<EMAIL_ADDRESS>
{esneminova<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we explore the capabilities of LLMs in capturing lexical-
semantic knowledge from WordNet on the example of the LLaMA-2-7b model and
test it on multiple lexical semantic tasks. As the outcome of our experiments,
we present TaxoLLaMA, the everything-in-one model, lightweight due to 4-bit
quantization and LoRA. It achieves 11 SotA results, 4 top-2 results out of 16
tasks for the Taxonomy Enrichment, Hypernym Discovery, Taxonomy Construction,
and Lexical Entailment tasks. Moreover, it demonstrates very strong zero-shot
performance on Lexical Entailment and Taxonomy Construction with no fine-
tuning. We also explore its hidden multilingual and domain adaptation
capabilities with a little tuning or few-shot learning. All datasets, code,
and model are available online.111https://github.com/VityaVitalich/TaxoLLaMA

##  1 Introduction

Recent studies in Natural Language Processing widely utilize Large Language
Models (LLMs) for their capability to store extensive knowledge (Sun et al.,
2023; Kauf et al., 2023; Tang et al., 2023) and to adapt quickly to different
tasks via in-context learning without backpropagation (Dong et al., 2023).
However, the application of LLMs to the classical lexical semantic tasks still
remains understudied: for instance, no recent experiments with LLMs have been
performed for the Hypernym Discovery task Camacho-Collados et al. (2018) for
different domains and languages. In Taxonomy Enrichment, LLMs are mostly used
to extract vector representations which are further processed with a complex
pipeline Jiang et al. (2022).

Figure 1: Training procedure of TaxoLLaMA: hypernym relations from the WordNet
are linearized and fed into the LLM model. The model aims at generating the
correct hypernym(s) as output.

Our work aims to investigate the capabilities of LLMs in addressing four tasks
requiring taxonomic knowledge: Hypernym Discovery, Taxonomy Enrichment,
Lexical Entailment, and Taxonomy Construction. We hypothesize that the model
fine-tuned with hypernym (IS-A relationships) would be useful for solving
various lexical semantic tasks. To verify this hypothesis, we outline a method
to compile a taxonomy-focused instruction tuning dataset, sourced from WordNet
(Miller, 1998), to bring the implicit word knowledge of an LLM to the
forefront when addressing lexical semantic tasks.

Having trained our model in this specialized setting, we are releasing the
TaxoLLaMA — the fine-tuned version of the LLaMA-2-7b model (Touvron et al.,
2023) — that is capable of solving tasks requiring taxonomic knowledge. Figure
1 presents the main idea of the model finetuning process. TaxoLLaMA operates
effectively in a zero-shot setting, surpassing SotA results in Lexical
Entailment tasks and Taxonomy Construction. With additional tuning, it also
achieves SotA performance in the Hypernym Discovery task across various
languages and in half of the Taxonomy Enrichment tasks. Furthermore, we have
optimized TaxoLLaMA to be lightweight through 4-bit quantization (Dettmers et
al., 2023) and the application of LoRA (Hu et al., 2021), making it feasible
to run on GPU devices with only 4.8Gb of GPU for forward pass and 5.5Gb for
fine-tuning, ensuring its accessibility for widespread use, e.g. using
Colab222https://colab.research.google.com.

(a) Lexical Semantic Tasks

(b) Generation and Ranking pipelines for Lexical Semantics tasks solution

Figure 2: Examples with input and output for each task are highlighted by
color. Rectangle “hypernym” denotes a word generated by the model; circle
means a node from the graph. Confidence score determines the existence of a
relationship between the two nodes provided in the input.

The contributions of the paper are as follows:

  * •

We introduce the use of LLMs across various lexical semantic tasks via
hypernym prediction and propose the appropriate taxonomy instruction tuning
method that exploits WordNet for dataset sampling.

  * •

We present a unified model —TaxoLLaMA — designed to address a broad spectrum
of lexical-sematic tasks achieving state-of-the-art (SotA) results in 11 out
of 16 tasks and secured the second rank in 4 tasks.

  * •

We present an instructive dataset based on WordNet-3.0 for training a
taxonomy-based LLM and collected definitions for input words in the Taxonomy
Enrichment datasets and the Lexical Entailment datasets using
Wikidata333http://wikidata.org and ChatGPT444https://chat.openai.com.

  * •

We perform a detailed error analysis for all tasks using both manual and
automatic approaches: e.g. we evaluate error patterns and model quality using
ChatGPT.

##  2 Related Work

In this section, we briefly describe previous approaches to the lexical
semantics tasks presented in the paper.

#### Hypernym Discovery

The recent study introduces a taxonomy-adapted, fine-tuned T5 model Nikishina
et al. (2023). Earlier models include the Recurrent Mapping Model (RMM) Bai et
al. (2021), which employs an MLP with residual connections and a contrastive-
like loss, and CRIM Bernier-Colborne and Barrière (2018), distinguished as the
best in SemEval, utilizing a similar MLP structure with a contrastive loss.
Additionally, the Hybrid model Held and Habash (2019) combines KNN with Hearst
patterns, while the 300 spasans Berend et al. (2018) offer an enhancement to
the traditional word2vec approach.

#### Taxonomy Enrichment

This task was addressed in SemEval-2016 Task 14 (Jurgens and Pilehvar, 2016),
aimed to construct a dataset and evaluation framework for taxonomy completion.
In more recent years, numerous different architectures have been proposed. TMN
(Zhang et al., 2021) exploits multiple scorers to find ⟨hypernym, hyponym⟩
pairs for a given query concept. The TaxoEnrich (Jiang et al., 2022) employs
two LSTMs (Staudemeyer and Morris, 2019) to encode ancestors and descendants
information. In addition, the TaxoExpan (Shen et al., 2020) uses GNN
(Scarselli et al., 2008) to predict whether the query is a child of an anchor
concept, while the GenTaxo model (Zeng et al., 2021) generates new concepts by
fusing sentence- and graph-based representations.

#### Taxonomy Construction

The taxonomy construction models include Graph2Taxo Shang et al. (2020), which
employs a sophisticated Graph Neural Network architecture, LMScorer, and
RestrictMLM Jain and Espinosa Anke (2022) use zero-shot RoBERTa or GPT2 for
pair relationship scoring, differing in their use of MASK or next token
probabilities. TAXI+ Aly et al. (2019) combines Hearst patterns with Poincaré
embeddings for refinement of the existing approaches.

#### Lexical Entailment

In recent lexical entailment models, LEAR Vulić and Mrkšić (2018) is a fine-
tuning method of transforming Euclidean space so that it reflects hyponymy-
hypernymy relations. In SeVeN Espinosa-Anke and Schockaert (2018) relations
between words are encoded. Pair2Vec Joshi et al. (2019) and GRV Jameel et al.
(2018) use words’ co-occurrence vectors and PMI. RELATIVE Camacho-Collados et
al. (2019) is a latent variable model that learns distributional relation
vectors. GBL (Hosseini et al., 2018) is GNN that utilizes "local" learning and
CTX Hosseini et al. (2021) is the improvement of CTX with contextual link-
prediction. The recent paper proposed etailment smoothing technique to the CTX
model resulted in SotA for the task McKenna et al. (2023).

##  3 Methodology

This section outlines the process of data collection and the subsequent
training of the model.

###  3.1 Data collection

We focus on hyponym-hypernym relationships sampled from the WordNet-3.0 graph
including both nouns and verbs. To prepare our training and validation sets,
we randomly pick edges to form pairs of hyponym-hypernym, the motivation for
precise choice is given in Section E. If a child node links to more than one
hypernym, we count each link as a separate pair. Additionally, we incorporate
definitions for child nodes from WordNet to disambiguate the sense of the
input word. As definitions may not be provided for some subtasks during
inference, we additionally generate definitions with ChatGPT for test sets
that lack pre-defined explanations or take them from Wikidata. The prompts for
such requests and the statistics of the generated definitions are presented in
the Appendix A in Examples LABEL:chatgpt_dataset-LABEL:chatgpt_dataset2 and in
Table 8. This step is highly required: the lack of definitions can
significantly reduce the performance of the model.

We include the following system prompt to describe the desired output
(LABEL:desired), the input with the definition (LABEL:inputformat), and the
target (LABEL:target):

desired [INST] <<SYS>> You are a helpful assistant. List all the possible words divided with a comma. Your answer should not include anything except the words divided by a comma <</SYS>> inputformat hyponym: tiger (large feline of forests in most of Asia having a tawny coat with black stripes) | hypernyms: [/INST] targetbig cat,

###  3.2 Training details

We introduce two versions of our model: TaxoLLaMA, the model trained on the
full WordNet-3.0 dataset for further community usage in lexical semantic
tasks, and TaxoLLaMA-b, designed for the benchmark tests. For this model, we
make sure that the training set does not include any nodes from the test sets
of those four tasks. The size of the training set for the first model is
$44772$ items, whereas the other model was fine-tuned with $36775$ samples.
The TaxoLLaMA-verb model, that we introduce in Section 5.4 trained on $7712$
samples. The main finetuning procedure of TaxoLLaMA and TaxoLLaMA-b is also
depicted in Figure 1.

To train in this setup, we use the LLaMA-2 model with $7$ billion parameters
Touvron et al. (2023). For better computational efficiency during training and
inference, we quantize the model to 4 bits and fine-tune it using a full-
precision LoRA adapter Dettmers et al. (2023). During pre-training, we used a
batch size of $32$ and a learning rate of $3e^{-4}$, applying a cosine
annealing scheduler. Any further fine-tuning for different domains or
languages was done with a batch size of $2$ and a learning rate of $3e^{-4}$,
without using schedulers. Other details are described in Appendix F.

| 1A: English | 2A: Medical; | 2B: Music
---|---|---|---
CRIM (Bernier-Colborne and Barrière, 2018) | 36.10 | 54.64 | 60.93
T5 (Nikishina et al., 2023) | 45.22 | 44.73 | 53.35
RMM (Bai et al., 2021) | 39.07 | 54.89 | 74.75
Hybrid (Held and Habash, 2019) | 34.07 | 64.47 | 77.24
TaxoLLaMA zero-shot | 38.05 | 43.09 | 42.7
TaxoLLaMA-b zero-shot | 37.66 | 42.2 | 44.36
TaxoLLaMA fine-tuned | 54.39 | 77.32 | 80.6
TaxoLLaMA-b fine-tuned | 51.59 | 73.82 | 78.63

Table 1: MRR performance on Hypernym Discovery for English.

| 1B: Italian | 1C: Spanish
---|---|---
300-sparsans (Berend et al., 2018) | 25.14 | 37.56
T5 (Nikishina et al., 2023) | 24.04 | 27.50
TaxoLLaMA zero-shot | 1.95 | 2.21
TaxoLLaMA-b zero-shot | 1.47 | 2.08
TaxoLLaMA fine-tuned | 51.58 | 57.44
TaxoLLaMA-b fine-tuned | 50.95 | 58.61

Table 2: MRR performance on Multilingual Hypernym Discovery.

(a) Fine-tuning

(b) Few-shot learning

Figure 3: Experiments for domain and language adaptation on the Hypernym
Discovery datasets.

###  3.3 Task Adaptation

We propose two methods for adapting LLMs, fine-tuned with the WordNet
instructive dataset, to downstream tasks. They are presented in Figure 2 where
2(a) presents four tasks we test our model on, while 2(b) depicts two
adaptation approaches.

Generative Approach involves directly applying the same procedure as used in
training. Given a hyponym, we use the model to generate a list of
corresponding hypernyms. We apply this approach to the Hypernym Discovery and
Taxonomy Enrichment datasets.

Ranking Approach involves evaluating the hypernymy relation using perplexity:
a lower score indicates a stronger relationship. Beyond assessing this
relationship, we can also evaluate the hyponymy relation by simply reversing
the hypernym and hyponym positions.

We apply this approach for the Taxonomy Construction and Lexical Entailment
datasets with slight modifications that will be described in the respective
sections 4.3 and 4.4 in more detail.

##  4 Experiments

In this section, we assess the proposed methodology and the fine-tuned models,
TaxoLLaMA and TaxoLLaMA-b, on four lexical semantic tasks: Hypernym Discovery,
Taxonomy Enrichment, Lexical Entailment, and Taxonomy Construction. We
evaluate models in a zero-shot setting and after fine-tuning on the provided
train sets for each task.

###  4.1 Hypernym Discovery

We test our model on the Hypernym Discovery task from SemEval-2018 Camacho-
Collados et al. (2018) using our generative approach, which involves
predicting a list of hypernyms for a given hyponym (see example in Figure
2(a)). This task features an English test set for general hypernyms and two
domain-specific “Music” and “Medical” sets, and general test sets for Italian
and Spanish. Performance is measured using the Mean Reciprocal Rank (MRR)
metric.

###  4.2 Taxonomy Enrichment

Taxonomy Enrichment aims to identify the most appropriate placement for a
missing node within a taxonomy. Continuing the approach of prior works Zhang
et al. (2021); Jiang et al. (2022), the goal was framed as ranking nodes from
the graph based on their likelihood of being the hypernym, where successfully
placing the node means ranking its correct hypernyms at the top. In our setup,
we use the generative approach described in Section 3.3 and depicted in Figure
2(b).

The Taxonomy Enrichment benchmark encompasses the WordNet Noun, WordNet Verb,
MAG-PSY, and MAG-CS datasets Jiang et al. (2022); Shen et al. (2020). To
ensure consistency, 1000 nodes from each dataset were sampled to match the
test set from TaxoExpan Shen et al. (2020). Following Jiang et al. (2022), we
consider scaled MRR Ying et al. (2018) as the main metric, which is the
regular MRR multiplied by 10 and averaged over all of a node’s correct
hypernyms.

| MAG-CS | MAG-PSY | Noun | Verb
---|---|---|---|---
TaxoExpan (Shen et al., 2020) | 19.3 | 44.1 | 39.0 | 32.5
GenTaxo (Zeng et al., 2021) | 23.9 | 46.4 | 28.6 | 42.8
TMN (Zhang et al., 2021) | 24.3 | 53.1 | 36.7 | 35.4
TaxoEnrich (Jiang et al., 2022) | 57.8 | 58.3 | 44.2 | 45.2
TaxoLLaMA zero-shot | 7.4 | 7.3 | n/a | n/a
TaxoLLaMA-b zero-shot | 8.5 | 6.6 | n/a | n/a
TaxoLLaMA fine-tuned | 24.9 | 29.8 | 48.0 | 52.4
TaxoLLaMA-b fine-tuned | 30.2 | 31.4 | 45.9 | 51.9

Table 3: Scaled MRR Across Tasks for Taxonomy Enrichment. Here, “n/a” stands
for “not applicable”, as TaxoLLaMA has already seen WordNet data and its
performance cannot be considered as zero-shot.

| S | E | F
---|---|---|---
TexEval-2 best (Bordea et al., 2015) | 31.30 | 30.00 | 36.01
TAXI+ (Aly et al., 2019) | 41.40 | 30.90 | 34.10
Graph2Taxo pure (Shang et al., 2020) | 39.00 | 37.00 | -
Graph2Taxo best (Shang et al., 2020) | 47.00 | 40.00 | -
LMScorer (Jain and Espinosa Anke, 2022) | 31.80 | 26.40 | 24.90
RestrictMLM (Jain and Espinosa Anke, 2022) | 37.90 | 23.00 | 24.90
TaxoLLaMA | 44.55 | 45.13 | 51.71
TaxoLLaMA-b | 42.36 | 44.82 | 51.18

Table 4: F1 score for the Taxonomy Construction Task. “S” stand for the
(S)cience dataset, “E” for the (E)nvironment dataset, and “F” stands for the
(F)ood domain dataset.

###  4.3 Taxonomy Construction

This task aims to assemble a taxonomy given a list of nodes and a root. For
this task, we employ datasets from Texeval-2 Bordea et al. (2015) with
“Eurovoc science”, “Eurovoc environment” and “WordNet food” subtasks. To gauge
the model performance, we apply the F1 measure likewise in the original
competition Bordea et al. (2015).

We evaluate our model with the ranking approach applied to all node pairs.
Using this principle, we iteratively established a threshold below which pairs
are considered to have a relationship. The threshold for the “Food” domain was
set to $1.8$, for “Environment” to $4.6$, and for “Science” to $1.89$. To
further refine the graph, we eliminate cycles by deleting the edge inside a
cycle with the highest perplexity. Additionally, we limit each node to a
maximum of three hypernyms. For nodes associated with more than three
hypernyms, only three with the lowest perplexity scores are retained.

###  4.4 Lexical Entailment

Lexical Entailment aims at identifying semantic relationships between phrase
pairs. We utilized the ANT entailment subset Guillou and de Vroe (2023) (a
detailed enhancement of the Levy/Holt dataset Holt (2019)) and Hyperlex
benchmark Vulić et al. (2017) for our experiments.

#### ANT dataset

To evaluate the entailment relations, we utilize ratio of hypernym and hyponym
ranking score, normalized via the L2 norm to represent the probability of
entailment. For instance, we calculate the perplexity for “move” as a hypernym
of “walk” ($PPL_{m\rightarrow w}$) and vice versa ($PPL_{w\rightarrow m}$).
The ratio $\frac{PPL_{m\rightarrow w}}{PPL_{w\rightarrow m}}$ of these scores
will thus indicate the model’s confidence.

#### HyperLex dataset

Hyperlex focuses on the entailment for verbs and nouns, evaluating on a scale
from 0 to 10. A score of 0 indicates no entailment, while 10 means strong
entailment. The goal is to achieve the highest correlation with the gold-
standard scores. For Hyperlex, we consider the ranking approach with no
additional processing.

Previous methods generate embeddings and train a simple SVM on the Hyperlex
training set. Fine-tuned models like RoBERTa demand substantial computational
efforts and are tailored to the Hyperlex dataset. Compared to those prior
studies, our zero-shot model uses perplexities directly as the predictions
without a need for training. Therefore, a direct comparison might overlook the
unique methodologies and resource implications, suggesting that each approach
should be evaluated within its specific context.

##  5 Results

This section describes the main results of generative and ranking setup
experiments for all tasks.

| AUCN | AP
---|---|---
GBL (Hosseini et al., 2018) | 3.79 | 58.36
CTX (Hosseini et al., 2021) | 15.44 | 65.66
GBL-PK=4 | 13.91 | 64.71
CTX-PK=4 | 25.86 | 67.47
TaxoLLaMA zero-shot | 0.89 | 51.61
TaxoLLaMA-b zero-shot | 2.82 | 54.24
TaxoLLaMA-verb zero-shot | 19.28 | 69.51

Table 5: Performance on the Lexical Entailment ANT dataset

Setting | Model | Lexical | Random
---|---|---|---
fine-tuned | RoBERTa best (Pitarch et al., 2023) | 79.4 | 82.8
RoBERTa mean (Pitarch et al., 2023) | 65.8 | 63.8
LEAR (Vulić and Mrkšić, 2018) | 54.4 | 69.2
zero-shot | Relative (Camacho-Collados et al., 2019) | 54.3 | 58.4
Pair2Vec (Joshi et al., 2019) | 33.4 | 54.3
GRV SI (Jameel et al., 2018) | 48.3 | 55.4
SeVeN (Espinosa-Anke and Schockaert, 2018) | 46.9 | 62.7
FastText | 43.9 | 54.3
TaxoLLaMA | 70.2 | 59.3

Table 6: Spearman Correlation for lexical and random test subsets of Hyperlex
benchmark.

###  5.1 Hypernym Discovery

The results for the English language, as shown in Table 2, indicate that both
the fine-tuned TaxoLLaMA and TaxoLLaMA-b significantly outperform previous
SotA results. While the zero-shot performance of our models may be lower than
when fine-tuned, they still deliver comparable outcomes to previous results
for general English tasks and do not fall far behind in domain-specific tasks.

#### Multilinual Performance

For Italian and Spanish, the fine-tuned model surpasses previous SotA results.
We might assume the model effectiveness in a multilingual setting, knowing
that LLaMA-2 is initially multilingual and that previous finetuning was
performed exclusively on English pairs. However, we observe that the zero-shot
performance struggles to generate accurate hypernyms for languages other than
English.

#### Zero-shot Performance

To investigate this zero-shot underperformance, we analyzed the effects of
fine-tuning on both domains and languages, as shown in Figure 3(a). It’s clear
that, except for task 2B, the model exceeds previous SotA results with just 50
samples for fine-tuning. Additionally, the fluctuating scores highlight the
model’s sensitivity to the quality and nature of the training data.

#### Few-shot Performance

We also explored the few-shot learning approach for languages to assess the
model adaptability in an in-context learning environment, as shown in Figure
3(b). The model surpassed previous SotA benchmarks with a near-logarithmic
pattern of improvement for the Italian language with 30 and 50 shots, yet not
performing as well for Spanish. We attribute the suboptimal few-shot
performance to the 4-bit quantization and its relatively small size.

###  5.2 Taxonomy Enrichment

The results presented in Table 4 show that TaxoL̃LaMA surpasses all previous
methods on the WordNet Noun and WordNet Verb datasets but does not perform as
well as the current SotA method on the more specialized MAG-CS and MAG-PSY
taxonomies even after fine-tuning. We also notice that TaxoLLaMA-b, having
less data, unexpectedly performed better on the MAG datasets. To delve deeper
into the reasons behind overall underperformance, we conducted a comprehensive
error analysis, detailed in Section 6.1.

###  5.3 Taxonomy Construction

The results in Table 4 demonstrate that applying our method directly leads to
SotA performance on the “Environment” and “Food” datasets, and secures a
second-place ranking for the “Science” dataset. Further analysis of the graphs
generated through our modeling is provided in Section 6.2.

###  5.4 Lexical Entailment

The results of TaxoLLaMA on the Lexical Entailment datasets surpassed our
expectations.

#### Results on the ANT Dataset

From the results on the ANT dataset in Table 6, we benchmark our models
against prior SotA performances. A notable finding was the significant
difference in performance between TaxoLLaMA, which was trained on both nouns
and verbs and TaxoLLaMA-verb, which focused solely on verbs.

TaxoLLaMA-verb outperformed TaxoLLaMA in Lexical Entailment, suggesting
difficulties in processing nouns and verbs simultaneously that might impede
verb learning, possibly due to quantization and LORA adapter tuning
constraints. This issue seems specific to the entailment task, as it does not
emerge in other tasks, such as Taxonomy Enrichment, which also includes a verb
dataset. This discrepancy could stem from metrics requiring precise normalized
perplexity ranking.

Table 6 shows that TaxoLLaMA-verb achieves SotA performance on Average
Precision and is second by normalized AUC. The comparison with previous SotA
results is skewed, as the best-performing models benefited from the use of
additional Entailment Smoothing McKenna et al. (2023) on top of the model.
This technique has yet to be applied to our models, which might be a promising
direction for future enhancements.

#### Results on the HyperLex Dataset

Table 6 demonstrates the superiority of our model over the previous SotA in a
zero-shot context for the “Lexical” subset and a second-place ranking for the
“Random” subset. Contrary to the common trend where other models score higher
on the random subset, our method does not follow this pattern, likely due to
its independence from the training data, suggesting that the larger training
size of the random subset benefits other methods more. Despite the
straightforward zero-shot approach of our model, it still achieves notably
high results. Future work could explore using this score as a meta-feature in
task-specific models or adapting our entire model more closely to this task.

Figure 4: Average percentage of error types across Hypernym Discovery and
Taxonomy Enrichment datasets.

##  6 Error Analysis

In this section, we analyze the errors made by the TaxoLLaMA model, explore
the reasons behind these inaccuracies, and suggest potential strategies for
mitigation of LLMs applied to taxonomies.

###  6.1 Hypernym Discovery and Taxonomy Enrichment

As we apply the same generative approach for both Hypernym Discovery and
Taxonomy Enrichment we perform the joint error analysis. We split the process
into four steps: (i) manual analysis to identify the most common errors; (ii)
automatic analysis of errors using ChatGPT; (iii) comparing and merging the
most common errors identified; (iv) classification of the errors using
ChatGPT.

First, we took about 200 random examples from both Hypernym Discovery and
Taxonomy Enrichment datasets and wrote explanations of why the model failed to
generate the correct hypernym. The following five classes were identified: (i)
predicted hypernyms are too broad; (ii) incorrect/irrelevant definition
generated by ChatGPT; (iii) the model was unable to generate relevant
candidates in the same semantic field; (iv) miscellaneous cases.

ChatGPT suggested errors, prompts, and outputs can be seen in Appendix A, in
Table 9, Examples LABEL:chatgpt_input and LABEL:chatgpt_output. We merged them
into the following types:

Figure 5: Automatic Evaluation of the MAG datasets with the ChatGPT model.
“True” denotes the number of gold answers that ChatGPT preferred over
TaxoLLaMA answers; “Predicted” is when ChatGPT preferred TaxoLLaMA output;
“Both” and “None” options were also possible answers for ChatGPT.

  1. 1.

Overly Broad Predictions: The model often generates predictions encompassing a
broader concept than the true hypernym.

  2. 2.

Underly Broad Predictions: Conversely, some predictions are too narrow and
fail to capture the broader concept represented by the true hypernym.

  3. 3.

Inaccurate Predictions: The model may predict words that are very semantically
close to the true hypernym but struggles with fitting into the exact wording

  4. 4.

Conceptual Ambiguity: The model may struggle with ambiguous input words or
concepts, leading to incorrect predictions.

  5. 5.

Incorrect definitions: The model gets confused with the incorrect/inaccurate
definition retrieved from external sources.

We used the prompt LABEL:class_chatgpt presented in Appendix A to classify
incorrectly predicted instances. The results for each task and each dataset
are presented in Appendix B in Table 10 and in Figure 4 for average
distribution. We also provide Table 11 with an example for each error type.
The most common issue (75% of cases) is overly broad predicted concepts. It
can be explained by the model adaptation to domain datasets that are richer
than WordNet, like the “Music” and “Medical” domains. For Italian and Spanish,
significant inaccuracies were attributed to grammatical complexities, due to
the dataset limitations, linguistic intricacies, and lack of pre-training
data. Similarly, MAG datasets faced issues with specificity and ambiguity,
which led to lower results of TaxoLLaMA compared with Wordnet-based ones, as
shown in Table 4.

Manual examination of MAG taxonomies revealed misaligned instances, like
“olfactory toxicity in fish” being classified as a hyponym of “neuroscience”.
Furthermore, we assessed the accuracy of the predicted hypernyms using
ChatGPT, inspired by contemporary research Rafailov et al. (2023). We provided
the inputs, predicted nodes, and ground truth nodes to ChatGPT, asking for
preference. As depicted in Figure 5, ChatGPT mostly prefers neither of the
answers and ground truth hypernyms only slightly more frequently than the
predicted ones. The example of the input query is presented in Appendix A in
Example LABEL:class_chatgpt2.

We also evaluated the overlap between MAG datasets with WordNet data,
discovering minimal correspondence. Only 5% of the nodes from the entire graph
are present in the WordNet graph, with just 2% of edges for CS and 4% for PSY
matching. For 92% of these, there is no path in the WordNet graph. Among the
remaining connections, we found that 28% in CS and 10% in PSY represent cases
where nodes are hypernyms of themselves. This also partially explains the low
results of TaxoLLaMA, as MAG datasets seem to differ drastically from the data
used for TaxoLLaMA training.

Finally, we visualized the embeddings, revealing a notable discrepancy between
predictions and ground truth in the MAG subsets—a contrast not seen with
WordNet. Detailed observations of this analysis are documented in Appendix C.

###  6.2 Taxonomy Construction

Our analysis of the predicted graphs across various domain datasets, based on
statistics from Table 7, revealed consistent error patterns. Generally, the
gold standard graphs feature fewer edges, except in the environment domain.
The model often omits entire clusters of nodes rather than individual ones:
about 30% of nodes in the graph constructed with TaxoLLaMA lack a path to
their actual parents, indicating they reside in separate components.

Metric | Science | Environment | Food
---|---|---|---
Original
# Nodes | 125 | 261 | 1486
# Edges | 124 | 261 | 1533
Constructed
# Nodes | 78 | 216 | 1132
# Edges | 71 | 507 | 1372
# Nodes Missing | 48 | 45 | 354
# Weak Components | 8 | 5 | 51
# Nodes w/o original hypernym | 4 | 5 | 39
# Nodes w/o path to original hypernym | 29 | 70 | 308
# Nodes w/ path to original hypernym | 44 | 140 | 784
Mean Distance to original hypernym | 1.02 | 1.15 | 1.06

Table 7: Statistics of original graph and the constructed graph with highest
F1 score. The lower part of the table corresponds to constructed graph
statistics

Nevertheless, existing paths are nearly flawless, suggesting the model is
performing either very accurately or completely off-target. The model assigns
high perplexity to certain paths which are further incorrectly excluded. This
tendency indicates a particular challenge with concepts that are neither too
specific nor too general but fall in the middle of the taxonomy.

The nature of perplexity as a relative metric contributes to this issue, as
edges may not exist as they surpass the perplexity threshold. Adjusting the
threshold introduces incorrect edges, urging us to consider alternative
approaches like using LLMs as embedders, to enhance graph prediction accuracy.

###  6.3 Lexical Entailment

Our examination of the ANT dataset showed it has nearly 3000 test samples but
only 589 unique verbs. This means that errors on one verb could be replicated
throughout the dataset. However, examining the overlap with WordNet revealed
only 7 verbs were found in the same form. Lemmatization increased the count to
338, but about 42% of unique verbs still were not found in WordNet. No paths
for the verbs presented in WordNet were found, which might have influenced
model performance on the task. Hyperlex demonstrated better statistics, with
nearly half of the words being unique and 88% found in WordNet. Only 27% of
pairs presented in the taxonomy, and 99% lacked a connecting path.

Perplexity-related errors showed high values for polysemous pairs (e.g.,
“spade is a type of card”) and low values for synonyms or paraphrases,
indicating a semantic closeness but no hypernymy relation. This points to the
model struggle with lexical diversity and ambiguity, emphasizing the need for
disambiguation abilities in entailment tasks. Additional analysis is available
in Appendix D.

##  7 Conclusion

In this paper, we introduce TaxoLLaMA— an LLM fine-tuned on WordNet-3.0,
capable of solving various lexical semantic tasks via hypernym prediction. It
achieved SotA results in 11 out of 16 tasks and secured the second position in
four tasks. Manual and ChatGPT error analysis shows that the most errors (75%)
are overly broad predicted concepts, because of overfitting on the WordNet
structure and inability to adapt to the target datasets. Additional
experiments also showed that definitions contribute a lot to the final scores
for Taxonomy Enrichment. Partially, this might be because definitions might
help to better disambiguate input words. Regarding error analysis, the most
difficult datasets are MAGs, as they differ drastically from the data used for
initial TaxoLLaMA training.

In future work, we plan to extend our research to other tasks and datasets and
to enhance the finetuning with an additional loss function based on semantic
similarity. Additionally, we will experiment with “unlearning” or selective
forgetting Chen and Yang (2023) with the unified TaxoLLaMA model.

## Limitations

We find the main limitation of our work as follows:

  * •

Nowadays, dozens of large pre-trained generative models exist and we report
results only on LLaMA-2. It may be that some other base models used could
further push the results, however, our preliminary studies showed that LLaMA-2
showed decent performance on hypernymy prediction. Moreover, our goal was also
to provide a lightweight model that could be of further research even with
limited resources. Finally, the research focused on the LLM application and
not on an exhaustive search of all LLM models.

  * •

We did not test the “Ranking” approach to the Taxonomy Enrichment dataset,
which would be also possible, as finding the most appropriate node for the
input word could also be seen as ranking. However, this dataset required more
time and resources than other datasets, and the first experiments showed lower
results. Therefore, we leave these experiments for further research.

  * •

Possible “hypernymy hallucination” may also be considered as a limitation:
apart from the generalization capabilities the model may overpredict types, or
even invent new words or semantic categories.

  * •

Another specificity of our model is its possible excessive concentration on
one meaning, which may result in the inability to generate a wider variety of
options.

  * •

We tried to be exhaustive, but we might not have covered all existing
taxonomy-related subtasks, which we leave out of the scope of the research.

## Ethical Statement

In our research, we employ advanced neural models like LlaMA-2, which have
been pre-trained on a diverse corpus, including user-generated content.
Although the creators of these models have endeavored to remove harmful or
biased data, it is important to recognize that some biases may still persist
in the model outputs.

This acknowledgment does not undermine the validity of our methods. We have
designed our techniques to be flexible, allowing them to be applied to
alternative pre-trained models that have undergone more rigorous debiasing
processes. To the best of our knowledge, aside from the challenge of
mitigating inherent biases, our work does not raise any additional ethical
concerns.

## References

  * Aly et al. (2019) Rami Aly, Shantanu Acharya, Alexander Ossa, Arne Köhn, Chris Biemann, and Alexander Panchenko. 2019.  Every child should have parents: a taxonomy refinement algorithm based on hyperbolic term embeddings.
  * Bai et al. (2021) Yuhang Bai, Richong Zhang, Fanshuang Kong, Junfan Chen, and Yongyi Mao. 2021.  Hypernym discovery via a recurrent mapping model.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2912–2921, Online. Association for Computational Linguistics.
  * Berend et al. (2018) Gábor Berend, Márton Makrai, and Péter Földiák. 2018.  300-sparsans at SemEval-2018 task 9: Hypernymy as interaction of sparse attributes.  In _Proceedings of the 12th International Workshop on Semantic Evaluation_ , pages 928–934, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bernier-Colborne and Barrière (2018) Gabriel Bernier-Colborne and Caroline Barrière. 2018.  CRIM at SemEval-2018 task 9: A hybrid approach to hypernym discovery.  In _Proceedings of the 12th International Workshop on Semantic Evaluation_ , pages 725–731, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bordea et al. (2015) Georgeta Bordea, Paul Buitelaar, Stefano Faralli, and Roberto Navigli. 2015.  SemEval-2015 task 17: Taxonomy extraction evaluation (TExEval).  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 902–910, Denver, Colorado. Association for Computational Linguistics.
  * Camacho-Collados et al. (2018) Jose Camacho-Collados, Claudio Delli Bovi, Luis Espinosa-Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vered Shwartz, Roberto Navigli, and Horacio Saggion. 2018.  SemEval-2018 task 9: Hypernym discovery.  In _Proceedings of the 12th International Workshop on Semantic Evaluation_ , pages 712–724, New Orleans, Louisiana. Association for Computational Linguistics.
  * Camacho-Collados et al. (2019) Jose Camacho-Collados, Luis Espinosa-Anke, Shoaib Jameel, and Steven Schockaert. 2019.  A latent variable model for learning distributional relation vectors.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 4911–4917. International Joint Conferences on Artificial Intelligence Organization.
  * Chen and Yang (2023) Jiaao Chen and Diyi Yang. 2023.  Unlearn what you want to forget: Efficient unlearning for llms.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 12041–12052. Association for Computational Linguistics.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.  A survey on in-context learning.
  * Espinosa-Anke and Schockaert (2018) Luis Espinosa-Anke and Steven Schockaert. 2018.  SeVeN: Augmenting word embeddings with unsupervised relation vectors.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2653–2665, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Guillou and de Vroe (2023) Liane Guillou and Sander Bijl de Vroe. 2023.  Ant dataset.
  * Held and Habash (2019) William Held and Nizar Habash. 2019.  The effectiveness of simple hybrid systems for hypernym discovery.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3362–3367, Florence, Italy. Association for Computational Linguistics.
  * Holt (2019) Xavier Holt. 2019.  Probabilistic models of relational implication.
  * Hosseini et al. (2018) Mohammad Javad Hosseini, Nathanael Chambers, Siva Reddy, Xavier R. Holt, Shay B. Cohen, Mark Johnson, and Mark Steedman. 2018.  Learning typed entailment graphs with global soft constraints.  _Transactions of the Association for Computational Linguistics_ , 6:703–717.
  * Hosseini et al. (2021) Mohammad Javad Hosseini, Shay B. Cohen, Mark Johnson, and Mark Steedman. 2021.  Open-domain contextual link prediction and its complementarity with entailment graphs.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2790–2802, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Jain and Espinosa Anke (2022) Devansh Jain and Luis Espinosa Anke. 2022.  Distilling hypernymy relations from language models: On the effectiveness of zero-shot taxonomy induction.  In _Proceedings of the 11th Joint Conference on Lexical and Computational Semantics_ , pages 151–156, Seattle, Washington. Association for Computational Linguistics.
  * Jameel et al. (2018) Shoaib Jameel, Zied Bouraoui, and Steven Schockaert. 2018.  Unsupervised learning of distributional relation vectors.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 23–33, Melbourne, Australia. Association for Computational Linguistics.
  * Jiang et al. (2022) Minhao Jiang, Xiangchen Song, Jieyu Zhang, and Jiawei Han. 2022.  Taxoenrich: Self-supervised taxonomy completion via structure-semantic representations.  In _Proceedings of the ACM Web Conference 2022_ , pages 925–934.
  * Joshi et al. (2019) Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019.  pair2vec: Compositional word-pair embeddings for cross-sentence inference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3597–3608, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jurgens and Pilehvar (2016) David Jurgens and Mohammad Taher Pilehvar. 2016.  Semeval-2016 task 14: Semantic taxonomy enrichment.  In _Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016)_ , pages 1092–1102.
  * Kauf et al. (2023) Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2023.  Event knowledge in large language models: the gap between the impossible and the unlikely.  _Cognitive Science_ , 47(11):e13386.
  * McKenna et al. (2023) Nick McKenna, Tianyi Li, Mark Johnson, and Mark Steedman. 2023.  Smoothing entailment graphs with language models.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 551–563, Nusa Dua, Bali. Association for Computational Linguistics.
  * Miller (1998) George A Miller. 1998.  _WordNet: An electronic lexical database_.  MIT press.
  * Nikishina et al. (2023) Irina Nikishina, Polina Chernomorchenko, Anastasiia Demidova, Alexander Panchenko, and Chris Biemann. 2023.  Predicting terms in IS-a relations with pre-trained transformers.  In _Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings)_ , pages 134–148, Nusa Dua, Bali. Association for Computational Linguistics.
  * Pitarch et al. (2023) Lucía Pitarch, Jordi Bernad, Lacramioara Dranca, Carlos Bobed Lisbona, and J. Gracia. 2023.  No clues good clues: out of context lexical relation classification.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.
  * Scarselli et al. (2008) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008.  The graph neural network model.  _IEEE transactions on neural networks_ , 20(1):61–80.
  * Shang et al. (2020) Chao Shang, Sarthak Dash, Md. Faisal Mahbub Chowdhury, Nandana Mihindukulasooriya, and Alfio Gliozzo. 2020.  Taxonomy construction of unseen domains via graph-based cross-domain knowledge transfer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2198–2208, Online. Association for Computational Linguistics.
  * Shen et al. (2020) Jiaming Shen, Zhihong Shen, Chenyan Xiong, Chi Wang, Kuansan Wang, and Jiawei Han. 2020.  Taxoexpan: Self-supervised taxonomy expansion with position-enhanced graph neural network.  In _Proceedings of The Web Conference 2020_ , pages 486–497.
  * Staudemeyer and Morris (2019) Ralf C Staudemeyer and Eric Rothstein Morris. 2019.  Understanding lstm–a tutorial into long short-term memory recurrent neural networks.  _arXiv preprint arXiv:1909.09586_.
  * Sun et al. (2023) Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2023.  Head-to-tail: How knowledgeable are large language models (llm)? aka will llms replace knowledge graphs?  _arXiv preprint arXiv:2308.10168_.
  * Tang et al. (2023) Raphael Tang, Xinyu Zhang, Jimmy Lin, and Ferhan Ture. 2023.  What do llamas really think? revealing preference biases in language model representations.  _arXiv preprint arXiv:2311.18812_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  LLaMA 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Vulić and Mrkšić (2018) Ivan Vulić and Nikola Mrkšić. 2018.  Specialising word vectors for lexical entailment.  In _Proceedings of NAACL-HLT_ , pages 1134–1145.
  * Vulić et al. (2017) Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017.  Hyperlex: A large-scale evaluation of graded lexical entailment.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018.  Graph convolutional neural networks for web-scale recommender systems.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining_ , KDD ’18. ACM.
  * Zeng et al. (2021) Qingkai Zeng, Jinfeng Lin, Wenhao Yu, Jane Cleland-Huang, and Meng Jiang. 2021.  Enhancing taxonomy completion with concept generation via fusing relational representations.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 2104–2113.
  * Zhang et al. (2021) Jieyu Zhang, Xiangchen Song, Ying Zeng, Jiaze Chen, Jiaming Shen, Yuning Mao, and Lei Li. 2021.  Taxonomy completion via triplet matching network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 4662–4670.

##  Appendix A Using ChatGPT for Defenition Generation and Automatic Error
Analysis

Here below are two different example prompts LABEL:chatgpt_dataset and
LABEL:chatgpt_dataset2 for ChatGPT for definition generation. The MAG PSY and
MAG CS datasets for Taxonomy Enrichment, as well as ANT and HyperLex datasets
for Lexical Entailment, do not possess definitions. Therefore, we developed
several prompts specifically for the two types of datasets. For the hypernym
prediction we want to have definitions for one input word, whereas for Lexical
Entailment we expect to generate definitions for two words simultaneously, as
they might be helpful for disambiguation. Table 8 represents statistics for
the generated definitions for the datasets.

chatgpt_dataset

    Write a definition for the word/phrase in one sentence.

    Example:
    Word: caddle
    Definition: act as a caddie and carry clubs for a player

    Word: eszopiclone 3 mg
    Definition:

chatgpt_dataset2

    Write a definition for Word 1 and Word 2. Each definition should be
    in one sentence. If a word is ambiguous, use the other word to
    disambiguate it.

    Example:
    Word 1: depression
    Word 2: melancholy
    Definition 1: a mental state characterized by a pessimistic sense of
    inadequacy and a despondent lack of activity
    Definition 2: a constitutional tendency to be gloomy and depressed

    Word 1: conflict
    Word 2: disagreement

Dataset | Total | Generated with ChatGPT | From Wikidata
---|---|---|---
MAG PSY | 23,156 | 12,823 | 10,333
MAG CS | 29,484 | 5,714 | 23,770
ANT | 5,933 | 5,933 | -
HyperLex | 2,307 | 2,307 | -
Table 8: Statistics on definitions generated with ChatGPT for different tasks.

Here is an example of the input to ChatGPT to automatically detect error types
for hypernym prediction (Example LABEL:chatgpt_input) and the model output
(Example LABEL:chatgpt_output). Furthermore, Example LABEL:class_chatgpt
demonstrates an example prompt for automatic classification.

chatgpt_input

    Here is the data in four columns separated with "\t":

    topic: psychologyΨinput word: social construction of schizophrenia
    true: psychiatry, predictions: psychosis, social, psychological,
    and philosophical aspects, social psychology, psych, schizophrenia,
    antipsychotic, psychotic, antipsychotic

    topic: psychologyΨinput word: use heroinΨtrue: methadone maintenance,
    predictions: heroin, opium tincture, heroin, opium, tincture, heroin,
    opium, and morphine, man, heroin, drug addiction

    ...

    topic: psychologyΨinput word: coordinated management of meaningΨ
    true: social psychology, predictions: communication disorders,
    language and literacy, and social studies, language and literacy,
    metaphor, metonymy, and politics, metonymy, metaphor

    There is a topic, input word, correct hypernym (true), and model prediction
    (predicted). Your task is to compare true answers and model predictions
    and summarize what are the main error types of the model.

chatgpt_output

    To summarize the main error types of the model’s predictions compared
    to the true hypernyms:

    1. Overly Broad Predictions: The model often generates predictions that
    encompass a broader concept than the true hypernym.
        For example:
            "medication effects" was predicted with terms like "drug effects",
            "drug interactions", and "medication", which are broader than
            the true hypernym "clinical psychology" and "psychiatry."

    2. Underly Broad Predictions: Conversely, some predictions are too narrow
    and fail to capture the broader concept represented by the true hypernym.
        For example:
            "schizoid personality" was predicted with terms like "personality
            disorders" and "personality theory," which are narrower than the
            true hypernym "psychosis."

    3. Inclusion of Unrelated Terms: In some cases, the model includes terms
    in its predictions that are not directly related to the input word or
    true hypernym.
        For example:
            "pirisudanol" was predicted with terms like "dopamine receptor
            d1" and "dopamine receptor d2", which are unrelated to the true
            hypernym "psychiatry".

    4. Repetition of Terms: The model occasionally repeats terms in its
    predictions, which might indicate redundancy or lack of diversity
    in its output.
        For example:
            "pirisudanol" was predicted with repeated terms like
            "piracetam" and "hydergine."

    5. Inadequate Coverage of Concepts: Some input words and true
    hypernyms receive predictions that lack comprehensive coverage of
    related concepts.
        For example:
            "result category" was predicted with terms like "item
            response theory" and "qualitative research," which might not
            fully encapsulate the scope of the true hypernym "mathematics
            education."

class_chatgpt

    You are a helpful linguist. Your task is to do error classification
    for a dataset for hypernym prediction in a general domain. Here are
    possible classes:

    1. Overly Broad Predictions: The model often generates predictions
    encompassing a broader concept than the true hypernym.
    2. Underly Broad Predictions: Conversely, some predictions are too
    narrow and fail to capture the broader concept represented by the
    true hypernym.
    3. Inaccurate Predictions: The model may predict words that are very
    semantically close to the true hypernym, but struggles with fitting
    into the exact wording
    4. Conceptual Ambiguity: The model may struggle with ambiguous
    (polysemantic/multivalued) input words or concepts, leading to
    incorrect predictions.
    5. Incorrect definitions: The model gets confused with the incorrect/
    inaccurate definition retrieved from external sources

    You will be given an input word/phrase, true hypernym, and
    candidate hypernyms. Please, return a Python dict of error classes
    {1: 1, 2: 5, 3: 1, ..., 100:3}) for all instances below:

    id: 1, input word: parathyroid_hormone, true hypernym: hormone,
    predicted: hormonal agent, hormon, hematopoietic growth factor,
    growth factor of the blood, growth regulator, growth substance, growth
    ...
    id: 100, input word: proofreader, true hypernym: printer, predicted:
    reader, audience, audience member, spectator, viewer, listener,
    listener-in, hearer, recipient, witness, watcher, observer

Here is the prompt Example LABEL:class_chatgpt2 for ChatGPT in order to
automatically evaluate TaxoLLaMA results, as manual analysis has shown that
the gold true answers from MAG PSY and MAG CS datasets might not be of a good
quality either. Therefore, ChatGPT was required to choose between the true
answer from the dataset and the predicted candidate.

class_chatgpt2

        Here are the words in the psychological domain. Your task is to
        choose hypernym which is more relevant given two options.
        Answer 1 / 2 / both / none

    Example:
    social construction of schizophrenia
    option 1: psychosis
    option 2: psychiatry
    Answer: 2

    abdominal air sac
    option 1: air sac
    option 2: trachea
    Answer:

Error Type |  Descripton
---|---
Overly Broad Predictions |  The model often generates predictions that encompass a broader concept than the true hypernym.
Underly Broad Predictions |  Some predictions are too narrow and fail to capture the broader concept represented by the true hypernym
Inclusion of Unrelated Terms |  In some cases, the model includes terms in its predictions that are not directly related to the input word or true hypernym.
Repetition of Terms |  The model occasionally repeats terms in its predictions, which might indicate redundancy or lack of diversity in its output.
Inadequate Coverage of Concepts |  Some input words and true hypernyms receive predictions that lack comprehensive coverage of related concepts
Semantic Shift |  The model might exhibit errors related to semantic shift, where the predicted terms are semantically related to the input word but do not accurately reflect the intended meaning or context.
Conceptual Ambiguity |  The model may struggle with ambiguous input words or concepts, leading to predictions that lack clarity or specificity.
Domain-Specific Knowledge |  Errors may arise due to a lack of domain-specific knowledge or understanding of specialized terminology.
Cultural or Contextual Bias |  The model’s predictions may be influenced by cultural or contextual biases inherent in the training data. This could lead to inaccuracies, especially when dealing with topics or concepts that vary across cultures or contexts.
Incomplete Understanding of Relationships |  The model may struggle to understand complex relationships between concepts, leading to inaccurate predictions.
Word Sense Disambiguation |  Errors may occur due to difficulties in disambiguating between different senses of a word.
Knowledge Gap |  The model’s predictions may reflect gaps in its knowledge or understanding of certain concepts, resulting in inaccurate or incomplete responses.
Table 9:  12 Error types made by TaxoLLaMA for hypernym prediction detected by
ChatGPT.

##  Appendix B Error Type Analysis

This section represents the error types distribution across different datasets
for hypernym prediction: Hypernym Discovery and Taxonomy Enrichment in Table
10. Moreover, in Table 11 we provide an example for each error type that was
classified by ChatGPT.

| 1A: English | 2A: Medical | 2B: Music | 1B: Italian | 1C: Spanish | MAG-CS | MAG-PSY | Noun | Verb
---|---|---|---|---|---|---|---|---|---
Error 1 | 72.49% | 93.75% | 100.0% | 54.69% | 49.08% | 66.48% | 85.43% | 81.45% | 73.39%
Error 2 | 2.61% | 0.00% | 0.0% | 10.03% | 10.62% | 5.40% | 1.40% | 4.10% | 2.58%
Error 3 | 9.44% | 1.25% | 0.0% | 27.83% | 12.09% | 13.21% | 11.20% | 6.05% | 21.03%
Error 4 | 15.46% | 5.00% | 0.0% | 7.44% | 18.21% | 10.94% | 0.70% | 6.64% | 0.43%
Error 5 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 3.98% | 1.26% | 1.76% | 2.58%

Table 10: Errors Type Distribution across subset datasets for hypernym prediction: Hypernym Discovery and Taxonomy Enrichment. Error Type |  Input |  True |  Predicted
---|---|---|---
Overly Broad Predictions |  weed |  vascular plant |  plant, plant life, vegetable, vegetation, vascular plant, phanerogam, green plant, green thing, plant of the class, plant of the division
Underly Broad Predictions |  audio |  auditory communication |  signal, transmission, transmission signal, communication, communication signal, message, message signal, information, information signal, data, data signal, notification, notification signal, signaling
Inaccurate Predictions |  unconventionality |  unorthodoxy |  unorthodoxiness, unorthodxy, unconformity, heterodoxy, dissidence, disobedience, disloyalty, dis, disaffection, dissent, disagreement
Conceptual Ambiguity |  giant red paintbrush |  indian paintbrush |  paintbrushrush, paint brush, larkspur, delphinium, liliaceous plant, liliid monocot, lilioid
Incorrect definitions |  advance directive status |  documentation |  advance directive, directive, decision support system, clinical decision support system, health information technology
Definition: A do-not-resuscitate order (DNR), also known as Do Not Attempt
Resuscitation (DNAR), Do Not Attempt Cardiopulmonary Resuscitation (DNACPR)
Table 11: Examples for each Error type made by TaxoLLaMA for hypernym
prediction detected by ChatGPT.

##  Appendix C Distribution Visualization for Taxonomy Enrichment

Figure 6: t-SNE plot of distributions of ground truth nodes and predicted
nodes for taxonomy enrichment tasks. Each point represents a node, embedded
with SentenceBert. Color represents ground truth or model predictions (we ran
2 predictions with different seeds)

In this section, we delve into the distribution patterns of ground truth and
model predictions within the embedding space of the SentenceBert model Reimers
and Gurevych (2019). To achieve this, we initiated two separate model runs,
each with a distinct seed, aiming to capture the model’s variability.
Subsequently, we extracted the predicted candidates and the ground truth
hypernyms, mapping them into the embedding space provided by SentenceBert. To
facilitate a clearer visual analysis, we condensed the embedding dimensions to
50 using Principal Component Analysis and then applied t-SNE to project these
dimensions onto two principal components for visualization.

The findings, illustrated in Figure 6, reveal a distinct pattern between
WordNet and the MAG subsets (MAG_CS and MAG_PSY). WordNet displays a
significant overlap between the gold standard and predictions, despite a few
outliers that are presumably lower-ranked candidates. Conversely, the MAG
subsets exhibit different behavior, forming two slightly overlapping clusters
in the embedding space, suggesting a divergence between predictions and ground
truths. Additionally, these subsets contain more outliers, indicating
instances where the model may have completely missed the accurate hypernym
sense. It’s important to consider, however, that the SentenceBert model’s
representations could contribute to these discrepancies, especially for
concepts that are not well-represented in its training data.

##  Appendix D Hyperlex Correlation Analysis

Figure 7: Correlation plot of the perplexion ranks with the annotator’s score
on Hyperlex test sets. The line over the dots is a trend found with linear
regression. * shows that correlation has a p-value lower than $1e^{-4}$.

We also examine correlations using traditional methods for both test sets
(refer to Figure 7). By overlaying the linear regression trend on the observed
data points, a distinct trend emerges. However, this trend is notably impacted
by outliers, particularly within the Random set. This observation aligns with
findings from taxonomy construction, highlighting the model’s challenges in
accurately handling middle nodes or pairs exhibiting moderate entailment
strength.

When analyzing gold scores ranging from 2 to 8, the Random set displays a lack
of discernible trend, underscoring the model’s inconsistency in this area. The
Lexical set shows a slightly better trend in that area. However, with both
sets pairs characterized by strong entailment or minimal entailment are more
accurately categorized. This distinction significantly enhances the overall
correlation, which leads to a promising correlation score.

##  Appendix E Hypernym motivation

Inspired by recent advancements in semantic analysis, particularly the work
(Nikishina et al., 2023) on hyponym prediction, our study shifts focus towards
hypernym prediction for several compelling reasons. First, predicting
hypernyms is crucial for tasks such as taxonomy enrichment and hypernym
discovery. Second, the formulation of a loss component for hypernym prediction
is more straightforward, as most entities typically have a single correct
hypernym, unlike hyponyms, where multiple valid options exist. This
necessitates either adjustments to the loss function or extensive dataset
collection and analysis.

Furthermore, our experimentation with various prompts revealed that the most
effective format is detailed in the main section. Alternate prompts, which
adopted a more narrative style (e.g., “Given a hyponym ’tiger’, the hypernym
for it is"), led to the model generating paragraphs instead of concise
hypernym lists. Adjustments to the system prompt failed to rectify this.
Notably, appending a comma to the end of the target sequence significantly
improved the model output, encouraging it to list hypernyms instead of
producing narrative text.

In addressing the disambiguation challenge, we experimented with incorporating
definitions or technical identifiers from WordNet into the prompts.
Definitions proved more effective, likely owing to the model pre-training on
textual data. Attempts to generate hypernyms with specific WordNet codes
resulted in the model appending the same numerical identifier to each hypernym
which also resulted in lower scores.

##  Appendix F Hyperparameter motivation

Our analysis revealed the model acute sensitivity to the learning rate and
scheduler settings. The feasibility of employing a high learning rate in the
primary study was contingent upon the use of the LORA adapter, which modulates
weights without significant alterations. However, during full model fine-
tuning, we faced instabilities, manifesting as either overfitting or
underfitting—highlighting the necessity for further technical exploration into
optimal hyperparameter configurations. Additionally, the implementation of
4-bit quantization requires careful learning rate selection, as this process
notably compresses the weight distribution, demanding strategies to
effectively recover the model knowledge thereafter.

In the fine-tuning process, we deliberately chose a smaller batch size to
better accommodate the model to datasets, which are often limited in sample
size. Contrary to our expectations, increasing the learning rate and batch
size did not yield improved performance; this outcome can primarily be
attributed to the reduced number of steps the model takes toward adapting to
the specific domain. This strategy, however, did not apply to WordNet pre-
training, where we observed differing trends.

Apart from certain instruction tuning methodologies, our approach does not
involve calculating loss including on the instruction itself. Instead, loss
calculation is confined solely to the target tokens.

The experiments utilized Nvidia A100 or Quadro RTX 8000 GPUs. Pre-training for
TaxoLLaMA and TaxoLLaMA-b spanned 6 GPU hours, while TaxoLLaMA-verb required
less than 1 hour. Fine-tuning for MAG subsets took 5 GPU hours, attributed to
the lengthy definitions. Fine-tuning for other datasets was completed in under
an hour.


<EMAIL_ADDRESS>aff1]Faculty of Mathematics and Informatics, Shumen
University, 115 ”Universitetska” str., 9712 Shumen, Bulgaria. aff2]Faculty of
Mathematics and Informatics, Sofia University, 5 ”James Bourchier” blvd., 1164
Sofia, Bulgaria.

[cor1]Corresponding author<EMAIL_ADDRESS>
# Measuring heavy-tailedness of distributions

Pavlina K. Jordanova  Monika P. Petkova  [  [

###### Abstract

Different questions related with analysis of extreme values and outliers arise
frequently in practice. To exclude extremal observations and outliers is not a
good decision, because they contain important information about the observed
distribution. The difficulties with their usage are usually related with the
estimation of the tail index in case it exists. There are many measures for
the center of the distribution, e.g. mean, mode, median. There are many
measures for the variance, asymmetry and kurtosis, but there is no easy
characteristic for heavy-tailedness of the observed distribution. Here we
propose such a measure, give some examples and explore some of its properties.
This allows us to introduce classification of the distributions, with respect
to their heavy-tailedness. The idea is to help and navigate practitioners for
accurate and easier work in the field of probability distributions.

Using the properties of the defined characteristics some distribution
sensitive extremal index estimators are proposed and their properties are
partially investigated.

##  1 INTRODUCTION

More than 90 years scientists look for appropriate way for handling outliers.
[Irwin(1925)], [McKay(1935)], [Nair(1948)] and [Dixon(1950), Dixon(1953)]
consider them mainly with respect to the deviations of the distribution of the
maxima of the sample from the one of the maxima of the normal distribution.
They discuss the effect of removing outliers and propose some techniques for
handling them. Further on some other tests for outliers appear, see e.g.
Grubbs’ test [Grubbs(1969)]. They still neglects the importance of the extreme
values, do not take into account the fact that the standard deviation does not
obligatory exists, especially in case of heavy tailed distributions, and
compare the observed variable with the appropriate normal one. Recently
[Klebanov(2016), Klebanov  _et al._(2017)Klebanov, Antoch, Karlova, and
Kakosyan, Klebanov, Kakosyan, and Karlova(2016)] reminded this topic. In 1978
Tukey et al. give different definitions for mild and extremal outliers
[Tukey(1977)] and box-plots [McGill, Tukey, and Larsen(1978)] via the
quartiles of the distribution and the inter-quartile range ($IQR$). Here we
make classification of the distributions, with respect to the heaviness of
their tails using the theoretical: quartiles $Q_{1},Q_{2},Q_{3}$, $IQR$, lower
inner fences ($I_{L}$), lower outer fences ($O_{L}$), upper inner fences
($I_{R}$) and upper outer fences ($O_{R}$).

Suppose $X_{1},X_{2},...,X_{n}$ are mutually independent observations of a
random variable (r.v.) $X$ with cumulative distribution function (c.d.f.)
$F(x)=\mathbb{P}(X\leq x)$, probability density function (p.d.f.) $f$ and
increasing order statistics $X_{(1,n)}\leq...\leq X_{(n,n)}$. There are many
different possibilities to define empirical $p$-quantiles, $p\in(0,1)$. See
e.g. [Parzen(1979), Hyndman and Fan(1996), Langford(2006)]. We use the
following one
$\hat{F}^{\leftarrow}(p)=X_{([(n+1)p],n)}+\\{(n+1)p-[(n+1)p]\\}\\{X_{([(n+1)p]+1,n)}-X_{([(n+1)p],n)}\\}$,
where $[a]$ means the integer part of $a$ and $\frac{1}{n+1}\leq
p\leq\frac{n}{n+1}$. Let $\hat{Q}_{1}$, $\hat{Q}_{2}$, $\hat{Q}_{3}$ be the
empirical quartiles of the observed r.v. and
$\hat{IQR}=\hat{Q}_{3}-\hat{Q}_{1}$ be the corresponding empirical IQR. We use
the concepts for empirical: lower inner fences
$\hat{I}_{L}=\hat{Q}_{1}-1.5\texttimes\hat{IQR}$, upper inner fences
$\hat{I}_{R}=\hat{Q}_{3}+1.5\texttimes\hat{IQR}$, lower outer fences
$\hat{O}_{L}=\hat{Q}_{1}-3\texttimes\hat{IQR}$, upper outer fences
$\hat{O}_{R}=\hat{Q}_{3}+3\texttimes\hat{IQR}$, mild and extreme outliers,
given e.g. in [Devore(2015), SEMATHECH(), Watkins, Scheaffer, and Cobb(2010)].
We call an observation mild outlier if it is outside the interval
$[\hat{Q}_{1}-1.5\hat{IQR};\hat{Q}_{3}+1.5\hat{IQR}]$ and inside the interval
$[\hat{Q}_{1}-3\hat{IQR};\hat{Q}_{3}+3\hat{IQR}]$. We call an observation
extreme outlier if it is outside the interval
$[\hat{Q}_{1}-3\hat{IQR};\hat{Q}_{3}+3\hat{IQR}]$. See Figure 1 and
[Devore(2015)].

Different questions related with analysis of outliers arise frequently in
practice. The difficulties with their usage are usually related with the
estimation of the tail index in case it exists. Recently the extreme value
theory develops techniques for handling them, but it mainly relies on the
second order condition (see e.g. [De Haan and Ferreira(2007)]). It seems to be
difficult to be checked, handled and understood from practitioners. Due to
luck of information about the distribution outside the range of the data, its
tail should be estimated via many characteristics. There are many measures for
the center of the distribution, e.g. mean, mode, median. There are measures
for the variance, asymmetry and kurtosis, but there is no enough
characteristics for measuring heaviness of the tails of the distribution. Here
we propose such measures and give some examples. All of them are invariant
with respect to shifting of the discussed r.v. This allows us to introduce
classification of the distributions, with respect to their heavy-tailedness.
Using the outliers we propose a relatively easy techniques to recognize the
tail of the distribution and to estimate its index of regular variation in
case it exists. The idea is to help and navigate practitioners for accurate
statistical diagnostics and easier work in the field of probability
distributions. This approach provides benchmarks only for recognizing the
tails of the observed distribution. For better fit we need to take into
account also the specific form of its center.

Figure 1: Empirical box-plot, together with the empirical inner and outer
fences.

##  2 CLASSIFICATION OF DISTRIBUTIONS WITH RESPECT TO THEIR HEAVY-TAILEDNESS

Following Tukey, under theoretical box-plot of a given c.d.f. $F$ we
understand the one on Figure 2. One of the possibilities to make a tentative
fitting of the observed distribution is to compare its empirical box-plot with
the theoretical box-plot of the tested distribution. However this approach is
not robust, especially for small samples. See e.g. [Devore(2015)]. The
presence of outliers in a sample of independent observations strongly depends
not only of the distributional type, but also from the sample size. Therefore
we classify the distributions with respect to their probabilities to have mild
or extreme outliers. First of all let us mention that all numerical
characteristics that we introduce are invariant with respect to shifting of
the r.v.

Figure 2: Theoretical box-plot, together with theoretical inner and outer
fences.

###  2.1 Classification of the distributions with respect to heaviness of
their left tails

Definition 1. We call a r.v. $X$ and its c.d.f. $F$, $p_{mL}(X)$-mild-heavy
left-tailed if

$P(Q_{1}(F)-3IQR(F)<X\leq Q_{1}(F)-1.5IQR(F))=p_{mL}(X).$

Having in mind this definition we introduce classification of the
distributions with respect to their mild left tail.

Definition 2. A r.v. $X$ and a r.v. $Y$ belong to one and the same
$p_{mL}$-mild-heavy left-tailed class if $p_{mL}(X)=p_{mL}(Y)$. See Figure 3,
b). A r.v. $X$ has lighter mild-heavy left tail than a r.v. $Y$ if
$p_{mL}(X)<p_{mL}(Y)$.

Let us note that $p_{mL}(X)=p_{mL}(Y)$ does not mean neither that the $X$ and
$Y$ belong to one and the same distributional type, nor that they have one and
the same mean or variance. But if $X=Y$ in distribution then
$p_{mL}(X)=p_{mL}(Y)$.

The $p_{mL}$ characteristic is invariant with respect to shifting. More
precisely, for all $c_{1}\in\mathbb{R}$ $p_{mL}(c_{1}+X)=p_{mL}(X).$

Table 1 presents a small part of the this classification, where
$c_{m}=\left(\log_{1-\frac{log\,3}{log\,4}}\frac{3}{5}\right)^{-1}\approx
3.08$,
$c_{e}=\left(\log_{1-\frac{log\,3}{log\,4}}\frac{3}{4}\right)^{-1}\approx
5.47$. The fact that $p_{mL}$ characteristic of all normal distributions is
approximately $0.0035$ in practice means that if we observe such a r.v. we
should expect 3 or 4 mild left outliers to appear in a sample of 1000
observations. Analogously we should expect to have around 34 or 35 mild left
outliers in a sample of 10000 observations and so on. All negative exponential
distributions have approximately $0.0203$-mild-heavy left tail. So, if we
observe 100 independent realizations of exponentially distributed r.v. we
should expect to have 2 mild left outliers.

Table 1: Classification of some of the distributions with respect to their mild-heavy left-taildeness.  Distribution | $p_{mL}$
---|---
$U(a,b),a<b;Gamma(\alpha,\beta);Pareto(\alpha,\delta);Frechet(\alpha),0<\alpha<c_{m}$ | $0$
$Frechet(\alpha),\alpha\in(c_{m},c_{e}]$ | $exp\\{-\left(2.5log^{-1/\alpha}(4)-1.5log^{-1/\alpha}\frac{4}{3}\right)^{-\alpha}\\}\approx 0$
$Frechet(\alpha),\alpha>c_{e}$ | $exp\\{-\left(2.5log^{-1/\alpha}(4)-1.5log^{-1/\alpha}\frac{4}{3}\right)^{-\alpha}\\}-$
|
$-exp\\{-\left(4log^{-1/\alpha}4-3log^{-1/\alpha}\frac{4}{3}\right)^{-\alpha}\\}\approx
0$
$Gumbel$ | $\approx 0.00000043$
$N(\mu,\sigma^{2})$ | $\approx 0.0035$
$Weibull^{-}(\alpha)$ | $exp\\{-(2.5log^{1/\alpha}4-1.5log^{1/\alpha}\frac{4}{3})^{\alpha}\\}-$
| $-exp\\{-(4log^{1/\alpha}4-3log^{1/\alpha}\frac{4}{3})^{\alpha}\\}$
$Weibull^{-}(2)$ | $\approx 0.0102$
$t(2)$ | $\approx 0.0266$
$t(1)$ | $\approx 0.0328$
$-Exp(\lambda)$ | $\approx 0.0339$
$Weibull^{-}(1)$ | $\approx 0.0389$
$Weibull^{-}(0.5)$ | $\approx 0.0495$

What about more extreme left outliers? See Figure 3, a).

Definition 3. We call a r.v. $X$ and its c.d.f. $F$, $p_{eL}(X)$-extremely
heavy left-tailed if

$P(X<Q_{1}(F)-3IQR(F))=p_{eL}(X).$

Figure 3: Relation between the plot of the p.d.f. of a r.v. $X$ with c.d.f.
$F$, $p_{mL}(X)$ and $p_{eL}(X)$.

Definition 4. We say that a r.v. $X$ and a r.v. $Y$ belong to one and the same
$p_{eL}$-extremely heavy left-tailed class if $p_{eL}(X)=p_{eL}(Y)$.
Analogously, we say that a r.v. $X$ has lighter extremely heavy left tail than
a r.v. $Y$ if $p_{eL}(X)<p_{eL}(Y)$.

Table 2 presents some examples of classification of distributions with respect
to their extremely heavy left tails. In order to explain the results let us
consider again the normal distribution. The value $p_{eL}\approx 0.0000012$
means that in case we have independent observations on such a r.v. we should
expect to have 1 or 2 left extreme outliers in a sample of $10^{6}$
observations. Analogously we should expect to have approximately 12 left
extreme outliers in a sample of $10^{7}$ observations and so on.

Table 2: Classification of some of the distributions with respect to their extremely heavy left-tailedness.  Distribution | $p_{eL}=F(O_{L})$
---|---
U(a, b); Gamma$(\alpha,\lambda)$; Pareto($\alpha$, $\delta$); Frechet($\alpha$), $0<\alpha<c_{e}$ | 0
Frechet($\alpha$), $\alpha\geq c_{e}$; Gumbel | $\approx 0$
N($\mu$, $\sigma^{2}$) | $\approx 0.0000012$
$Weibull^{-}(\alpha)$ | $exp\\{-(4log^{1/\alpha}4-3log^{1/\alpha}\frac{4}{3})^{\alpha}$
$Weibull^{-}(2)$ | $\approx 0.0000668$
$-Exp(\lambda)$ | $\approx 0.0093$
$Weibull^{-}(1)$ | $\approx 0.0093$
$t(2)$ | $\approx 0.0146$
$t(1)$ | $\approx 0.0452$
$Weibull^{-}(0.5)$ | $\approx 0.0654$

Note: 1\. $p_{mL}(X)<p_{mL}(Y)$ is not equivalent to $p_{eL}(X)<p_{eL}(Y)$.

2\. If $p_{eL}(X)=p_{eL}(Y)$ or $p_{mL}(X)=p_{mL}(Y)$, this does not
obligatory mean that $X$ and $Y$ coincide in distribution.

3\. $p_{eL}(c_{1}+X)=p_{eL}(X)$, for all $c_{1}\in\mathbb{R}$.

###  2.2 Classification of the distributions with respect to heaviness of
their right tails

Analogously to the previous subsection we can work with the right tails. See
Figure 4, a) and b).

Definition 5. We call a r.v. $X$ and its c.d.f. $F$, $p_{mR}(X)$-mild-heavy
right-tailed if

$P(Q_{3}(F)+1.5IQR(F)<X\leq Q_{3}(F)+3IQR(F))=p_{mR}(X).$

Definition 6. We say that a r.v. $X$ and a r.v. $Y$ belong to one and the same
$p_{mR}$-mild-heavy right-tailed class if $p_{mR}(X)=p_{mR}(Y)$. A r.v. $X$
has lighter mild-heavy right tail than a r.v. $Y$ if $p_{mR}(X)<p_{mR}(Y)$.

Definition 7. We call a r.v. $X$ and its c.d.f. $F$, $p_{eR}(X)$-extremely
heavy right-tailed if

$P(X>Q_{3}(F)+3IQR(F))=p_{eR}(X).$

Figure 4: Relation between the plot of the p.d.f. of a r.v. $X$ with c.d.f.
$F$, $p_{mR}(X)$ and $p_{eR}(X)$.

Definition 8. A r.v. $X$ and a r.v. $Y$ belong to one and the same
$p_{eR}$-extremely heavy right-tailed class if $p_{eR}(X)=p_{eR}(Y)$. We say
that a r.v. $X$ has lighter extreme right tail than a r.v. $Y$ if
$p_{eR}(X)<p_{eR}(Y)$.

The properties of these characteristics are analogous to the corresponding one
of the left tails.

Some examples are given in Table 3. Again we observe that
$p_{mR}(X)<p_{mR}(Y)$ is not equivalent to $p_{eR}(X)<p_{eR}(Y)$. The analysis
is analogous to those made above for the left tails. It is well known that if
we consider only a fixed distribution with regularly varying tail, the bigger
the value of $\alpha$ the lighter the corresponding tail of the distribution
is. However when we consider the extremely heavy tails, which one of Pareto or
Frechet distribution has heavier right tail depends on their parameters. If
$X\sim Pareto(2,1)$ and $Y\sim Frechet(\alpha)$, $\alpha\geq 1$ then $X$ has
heavier right tail than $Y$, but $Frechet(0.5)$ has heavier extremal right
tail than $Pareto(0.5,1)$.

Table 3: Classification of some of the distributions with respect to heaviness of their right tails.  Distribution | $p_{mR}$ | $p_{eR}=\bar{F}_{X}(O_{R})$
---|---|---
$U(a,b),a<b,Weibull^{-}(\alpha),\,a,b\in\mathbb{R}$ | $0$ | $0$
$N(\mu,\sigma^{2})$ | $\approx 0.0035$ | $\approx 0.0000012$
$Gamma(2,\lambda)$, $\lambda>0$ | $\approx 0.0011$ | $\approx 0.000071$
$Gumbel$ | $\approx 0.0243$ | $\approx 0.0026$
$Exp(\lambda),\,\lambda>0$ | $\approx 0.0339$ | $\approx 0.0093$
$t(2)$ | $\approx 0.0266$ | $\approx 0.0146$
$Gamma(0.5,\lambda)$, $\lambda>0$ | $\approx\mathbf{0.0502}$ | $\approx 0.0255$
$Frechet(\alpha)$ | $1-exp^{-(2.5\sqrt[\alpha]{3.48}-1.5\sqrt[\alpha]{0.72})^{-\alpha}}$ | $1-e^{-(4\sqrt[\alpha]{3.48}-3\sqrt[\alpha]{0.72})^{-\alpha}}$
$Frechet(2)$ | $\approx 0.0429$ | $\approx 0.0406$
$t(1)$ | $\approx 0.0328$ | $\approx 0.0452$
$Pareto(\alpha,\delta)$ | $\frac{\delta^{-\alpha}}{4}(2.5-1.5\sqrt[\alpha]{\frac{1}{3}})^{-\alpha}-p_{eR}$ | $\frac{\delta^{-\alpha}}{4}(4-3\sqrt[\alpha]{\frac{1}{3}})^{-\alpha}$
$Pareto(2,1)$ | $\approx\mathbf{0.045}$ | $\approx 0.0486$
$Frechet(1)$ | $\approx 0.0415$ | $\approx 0.0817$
$Pareto(1,1)$ | $\approx 0.0417$ | $\approx 0.0833$
$Pareto(0.5,1)$ | $\approx 0.0331$ | $\approx 0.1306$
$Frechet(0.5)$ | $\approx 0.0323$ | $\approx 0.1360$

Note that if $X\sim Gamma(\alpha,\lambda)$, $\lambda>0$, then $p_{mL}(X)$,
$p_{eL}(X)$, $p_{mR}(X)$ and $p_{eR}(X)$ does not depend on $\lambda$.

###  2.3 Classification of the distributions with respect to heaviness of
their two-sided tails

Here, for the seek of completeness, we consider the two-sided heavy-tailedness
of the distributions. However in practice it is better to make a more detailed
comparison of the probabilities to have one-sided left or right, mild or
extreme outliers. It gives us more comprehensive picture about the tail
behaviour of the observed distribution.

Definition 9. We call a r.v. $X$ and its c.d.f. $F$, $p_{m2}(X)$-mild-heavy
two-tailed if

$P(Q_{1}(F)-3IQR(F)<X\leq Q_{1}(F)-1.5IQR(F)\cup Q_{3}(F)+1.5IQR(F)<X\leq
Q_{3}(F)+3IQR(F))=p_{m2}(X).$

Definition 10. A r.v. $X$ and a r.v. $Y$ belong to one and the same
$p_{m2}$-mild-heavy two-tailed class if $p_{m2}(X)=p_{m2}(Y)$. A r.v. $X$ with
c.d.f. $F$ has lighter mild two-tails than a r.v. $Y$ if
$p_{m2}(X)<p_{m2}(Y)$.

Definition 11. A r.v. $X$ and its c.d.f. $F$ are called $p_{e2}(X)$-extremely
heavy two-tailed if

$P(X<Q_{1}(F)-3IQR(F)\cup X>Q_{3}(F)+3IQR(F))=p_{e2}(X).$

Definition 12. A r.v. $X$ and a r.v. $Y$ belong to one and the same
$p_{e2}$-extremely heavy two-tailed class if $p_{e2}(X)=p_{e2}(Y)$ and a r.v.
$X$ has lighter extreme two-tails than a r.v. $Y$ if $p_{e2}(X)<p_{e2}(Y)$.

Note: Again the equalities $p_{m2}(X)=p_{m2}(Y)$ or $p_{e2}(X)=p_{e2}(Y)$,
does not obligatory mean that $X\stackrel{{\scriptstyle\rm d}}{{=}}Y$.

In Table 4 we have presented the values of $p_{m2}(X)$ and $p_{e2}(X)$ for
some probability laws. See Figure 5, a) and b).

Figure 5: Relation between the plot of the p.d.f. of a r.v. $X$, $p_{m2}(X)=\frac{p_{mL}(X)+p_{mR}(X)}{2}$ and $p_{e2}(X)=\frac{p_{eL}(X)+p_{eR}(X)}{2}$.  Table 4: Classification of some of the distributions with respect to heaviness of their two-sided tails.  Distribution | $p_{m2}(X)=p_{mL}(X)+p_{mR}(X)$ | $p_{e2}(X)=F_{X}(O_{L})+\bar{F}_{X}(O_{R})$
---|---|---
U(a, b) | 0 | 0
N($\mu$, $\sigma^{2}$) | $\approx 0.007$ | $\approx 0.000002$
$Gamma(2,\lambda)$, $\lambda>0$ | $\approx 0.0011$ | $\approx 0.000071$
$Weibull^{-}(\alpha)$ | $exp\\{-(2,5log^{1/\alpha}4-1,5log^{1/\alpha}\frac{4}{3})^{\alpha}\\}-p_{1}$ | $p_{1}=exp\\{-(4log^{1/\alpha}4-3log^{1/\alpha}\frac{4}{3})^{\alpha}\\}$
$Weibull^{-}(2)$ | $\approx 0.0102$ | $\approx 0.000067$
$Gumbel$ | $\approx 0.0243$ | $\approx 0.0026$
$-Exp(\lambda),\,Exp(\lambda),\,\lambda>0$ | $\approx 0.0339$ | $\approx 0.0093$
$Weibull^{-}(1)$ | $\approx 0.0388$ | $\approx 0.0093$
$Gamma(0.5,\lambda)$, $\lambda>0$ | $\approx\mathbf{0.0501}$ | $\approx 0.0255$
$t(2)$ | $\approx\mathbf{0.0532}$ | $\approx 0.0293$
$Frechet(\alpha)$ | $1-exp^{-(2.5\sqrt[\alpha]{3.48}-1.5\sqrt[\alpha]{0.72})^{-\alpha}}-p_{3}$ | $p_{3}=1-e^{-(4\sqrt[\alpha]{3.48}-3\sqrt[\alpha]{0.72})}$
$Frechet(2)$ | $\approx 0.0429$ | $\approx 0.0406$
$Pareto(\alpha,\delta)$ | $\frac{\delta^{-\alpha}}{4}(2.5-1.5\sqrt[\alpha]{\frac{1}{3}})^{-\alpha}-p_{2}$ | $p_{2}=\frac{\delta^{-\alpha}}{4}(4-3\sqrt[\alpha]{\frac{1}{3}})^{-\alpha}$
$Pareto(2,1)$ | $\approx 0.045$ | $\approx 0.0486$
$Weibull^{-}(0.5)$ | $\approx 0.0495$ | $\approx 0.0654$
$Frechet(1)$ | $\approx 0.0415$ | $\approx 0.0817$
$Pareto(1,1)$ | $\approx 0.0417$ | $\approx 0.0833$
$t(1)$ | $\approx\mathbf{0.0656}$ | $\approx 0.0903$
$Pareto(0.5,1)$ | $\approx 0.0331$ | $\approx 0.1306$
$Frechet(0.5)$ | $\approx 0.0323$ | $\approx 0.1360$

###  2.4 Algorithm for applications

Considering the outliers in a sample and comparing their relative frequencies
with $p_{mL}$, $p_{eL}$, $p_{mR}$ and $p_{eR}$ we are able to make a better
modelling of the tails of the distribution of the observed r.v. The algorithm
is the following:

  1. 1.

Determine $\hat{Q}_{1}$, $\hat{Q}_{2}$, $\hat{Q}_{3}$, $I\hat{Q}R$,
$\hat{I}_{L}$, $\hat{O}_{L}$, $\hat{I}_{R}$, $\hat{O}_{R}$ and compare the
empirical box-plot with the theoretical box-plot of the chosen distributions.

  2. 2.

Determine the relative frequencies of the left and right, mild and extreme
outliers.

  3. 3.

Make confidence intervals, based on the relative frequencies of mild or
extreme outliers. Compare these relative frequencies with $p_{mL}$ and
$p_{mR}$ in the list of distributions and chose appropriate classes of
distributions for modelling the probability law of the observed r.v.

  4. 4.

Make confidence intervals, based only on the relative frequencies of extreme
outliers. Compare these relative frequencies with $p_{eL}$ and $p_{eR}$ in the
list of distributions chosen in 3. and find the most appropriate
distributional types for modeling the observed r.v.

  5. 5.

Estimate the parameters of the chosen distributions.

  6. 6.

Use some goodness of fit test to chose the best model.

##  3 FIVE NEW ESTIMATORS OF THE EXTREMAL INDEX. EMPIRICAL STUDY.

In this section we suppose that $\widehat{Q}_{1}>0,\widehat{Q}_{1}\not=1$ and
at least one of the following two conditions hold:
$\widehat{Q}_{1}\not=\widehat{Q}_{3}$ or $\widehat{O}_{R}>1$. We propose to
model the observed r.v. with appropriate distribution with regularly varying
tail, i.e. such that $\lim_{y\to\infty}\frac{1-F(xy)}{1-F(y)}=x^{-\alpha}$ and
present five distribution sensitive estimators of the parameter $\alpha$. The
relative frequency $\hat{p}_{eR}$ of the right extreme outliers in the sample
is a strongly consistent and unbiased estimator of $p_{eR}$. The right outer
fence $\hat{O}_{R}$ is an asymptotically consistent estimator of the
theoretical $O_{R}$.

The following two estimators have very fast rate of convergence in case when
the observed r.v. is $Pareto(\alpha)$ distributed. See the empirical study and
Table 5.

$\hat{\alpha}_{Par,n}=-\frac{log\,\widehat{p}_{eR}}{log\,\widehat{O}_{R}},\quad\hat{\alpha}_{Q,Par,n}=\frac{log(3)}{log\widehat{Q}_{3}-log\widehat{Q}_{1}}.$

They have approximately the same properties as the Hill and the t-Hill
estimators.

Table 5: Empirical results.  Distribution | $n$ | $\hat{\alpha}_{Par,n}$ | $\hat{\alpha}_{Q,Par,n}$ | $\hat{\alpha}_{Frech,n}$ | $\hat{\alpha}_{Q,Frech,n}$ | $\hat{\alpha}_{Q,HillH,n}$ | The best
---|---|---|---|---|---|---|---
of $\xi$. | Mean | St. Dev. | Mean | St. Dev. | Mean | St. Dev. | Mean | St. Dev. | Mean | St. Dev. | estimator
$Pareto(\alpha)$ | $\alpha=0.5$ | $30$ | 0.5463 | 0.1356 | 0.5116 | 0.1442 | 0.531 | 0.138 | 0.7323 | 0.2064 | 1.2437 | 29.8589 | $\hat{\alpha}_{Q,Par,n}$
$100$ | 0.5119 | 0.0669 | 0.5035 | 0.0756 | 0.4954 | 0.0683 | 0.7207 | 0.1083 | 0.3904 | 74.1974 | $\hat{\alpha}_{Q,Par,n}$
$1000$ | 0.5015 | 0.0204 | 0.5005 | 0.0238 | 0.4846 | 0.0208 | 0.7164 | 0.034 | 1.7906 | 0.3105 | $\hat{\alpha}_{Q,Par,n}$
$10000$ | 0.5001 | 0.0063 | 0.5002 | 0.0074 | 0.4831 | 0.0063 | 0.716 | 0.0106 | 1.761 | 0.0922 | $\hat{\alpha}_{Par,n}$
$\alpha=1$ | $30$ | 1.0671 | 0.2528 | 1.026 | 0.2869 | 1.05 | 0.2568 | 1.4685 | 0.4106 | -2.1133 | 17.9816 | $\hat{\alpha}_{Q,Par,n}$
$100$ | 1.0312 | 0.1486 | 1.0092 | 0.1516 | 1.0141 | 0.1518 | 1.4446 | 0.2171 | -3.0747 | 9.8974 | $\hat{\alpha}_{Q,Par,n}$
$1000$ | 1.0022 | 0.0422 | 1.0006 | 0.0469 | 0.9848 | 0.0432 | 1.4322 | 0.0671 | -2.3396 | 0.2569 | $\hat{\alpha}_{Q,Par,n}$
$10000$ | 1.0004 | 0.0132 | 1 | 0.0146 | 0.983 | 0.0136 | 1.4314 | 0.021 | -2.324 | 0.081 | $\hat{\alpha}_{Q,Par,n}$
$\alpha=2$ | $30$ | 1.9996 | 0.4313 | 2.0514 | 0.566 | 1.9789 | 0.4351 | 2.9364 | 0.8102 | -1.134 | 0.2572 | $\hat{\alpha}_{Par,n}$
$100$ | 2.0772 | 0.3332 | 2.0185 | 0.3021 | 2.0607 | 0.338 | 2.8893 | 0.4325 | -1.0865 | 0.0938 | $\hat{\alpha}_{Q,Par,n}$
$1000$ | 2.0083 | 0.0937 | 2.0021 | 0.094 | 1.9919 | 0.0953 | 2.8658 | 0.1346 | -1.0766 | 0.0273 | $\hat{\alpha}_{Q,Par,n}$
$10000$ | 2 | 0.0292 | 1.9998 | 0.03 | 1.9836 | 0.0297 | 2.8624 | 0.043 | -1.0737 | 0.0087 | $\hat{\alpha}_{Par,n}$
$Frechet(\alpha)$ | $\alpha=0.5$ | $30$ | 0.5738 | 0.1549 | 0.354 | 0.0826 | 0.5568 | 0.1564 | 0.5067 | 0.1182 | 0.9214 | 9.8233 | $\hat{\alpha}_{Q,Frech,n}$
$100$ | 0.5335 | 0.0738 | 0.3508 | 0.0442 | 0.5151 | 0.0747 | 0.5021 | 0.0632 | 0.7301 | 0.2186 | $\hat{\alpha}_{Q,Frech,n}$
$1000$ | 0.5201 | 0.0222 | 0.3494 | 0.0139 | 0.5014 | 0.0225 | 0.5001 | 0.0199 | 0.7012 | 0.0564 | $\hat{\alpha}_{Q,Frech,n}$
$10000$ | 0.519 | 0.007 | 0.3494 | 0.0044 | 0.5002 | 0.007 | 0.5001 | 0.0062 | 0.6989 | 0.0175 | $\hat{\alpha}_{Q,Frech,n}$
$\alpha=1$ | $30$ | 1.0825 | 0.2599 | 0.7066 | 0.1648 | 1.0654 | 0.2637 | 1.0115 | 0.2358 | 3.2049 | 324.4615 | $\hat{\alpha}_{Q,Frech,n}$
$100$ | 1.0506 | 0.1534 | 0.7023 | 0.0892 | 1.0337 | 0.1565 | 1.0053 | 0.1277 | 5.5908 | 707.2482 | $\hat{\alpha}_{Q,Frech,n}$
$1000$ | 1.0209 | 0.0445 | 0.6989 | 0.0276 | 1.0038 | 0.0454 | 1.0004 | 0.0396 | -15.003 | 1031.205 | $\hat{\alpha}_{Q,Frech,n}$
$10000$ | 1.0174 | 0.014 | 0.6987 | 0.0088 | 1.0003 | 0.0143 | 1.0001 | 0.0126 | 287.2494 | 14959.87 | $\hat{\alpha}_{Q,Frech,n}$
$\alpha=2$ | $30$ | 1.9469 | 0.3957 | 1.4131 | 0.3275 | 1.9289 | 0.3988 | 2.0227 | 0.4688 | -1.5699 | 1.755 | $\hat{\alpha}_{Q,Frech,n}$
$100$ | 2.094 | 0.3418 | 1.4031 | 0.178 | 2.081 | 0.3461 | 2.0084 | 0.2548 | -1.4384 | 0.1996 | $\hat{\alpha}_{Q,Frech,n}$
$1000$ | 2.0202 | 0.0978 | 1.3975 | 0.0554 | 2.0072 | 0.0993 | 2.0004 | 0.0794 | -1.4013 | 0.0553 | $\hat{\alpha}_{Q,Frech,n}$
$10000$ | 2.0144 | 0.0309 | 1.3974 | 0.0175 | 2.0014 | 0.0313 | 2.0002 | 0.025 | -1.3975 | 0.0176 | $\hat{\alpha}_{Q,Frech,n}$
$Hill-$ | $\alpha=0.5$ | $30$ | 0.4692 | 0.1223 | 0.2921 | 0.0675 | 0.4534 | 0.1234 | 0.4181 | 0.0966 | 2.164 | 0.7827 | $\hat{\alpha}_{Par,n}$
$100$ | 0.4393 | 0.0591 | 0.2925 | 0.0367 | 0.4221 | 0.0598 | 0.4186 | 0.0525 | 0.5647 | 0.9664 | $\hat{\alpha}_{Par,n}$
$1000$ | 0.4286 | 0.0178 | 0.2915 | 0.0115 | 0.411 | 0.0179 | 0.4173 | 0.0165 | 0.5012 | 0.0342 | $\hat{\alpha}_{Q,HillH,n}$
$10000$ | 0.4278 | 0.0057 | 0.2915 | 0.0037 | 0.4102 | 0.0057 | 0.4172 | 0.0052 | 0.5001 | 0.0107 | $\hat{\alpha}_{Q,HillH,n}$
$\alpha=1$ | $30$ | 0.8051 | 0.205 | 0.4134 | 0.0926 | 0.788 | 0.2077 | 0.5917 | 0.1325 | 2.6214 | 187.5964 | $\hat{\alpha}_{Par,n}$
$100$ | 0.7581 | 0.1056 | 0.4127 | 0.0505 | 0.7403 | 0.1075 | 0.5908 | 0.0723 | 1.0795 | 0.4421 | $\hat{\alpha}_{Q,HillH,n}$
$1000$ | 0.7377 | 0.0315 | 0.4115 | 0.0158 | 0.7194 | 0.0321 | 0.589 | 0.0227 | 1.0055 | 0.0948 | $\hat{\alpha}_{Q,HillH,n}$
$10000$ | 0.7355 | 0.0098 | 0.4113 | 0.005 | 0.7172 | 0.01 | 0.5887 | 0.0072 | 1.0008 | 0.0908 | $\hat{\alpha}_{Q,HillH,n}$
$Horror(\alpha)$ | $\alpha=2$ | $30$ | 1.2185 | 0.283 | 0.5198 | 0.1143 | 1.2019 | 0.2859 | 0.744 | 0.1636 | 1.543 | 69.0628 | $\hat{\alpha}_{Q,HillH,n}$
| $100$ | 1.1965 | 0.1785 | 0.519 | 0.0631 | 1.1812 | 0.1816 | 0.7429 | 0.0903 | 2.8577 | 28.6543 | $\hat{\alpha}_{Q,HillH,n}$
| $1000$ | 1.1554 | 0.05 | 0.5178 | 0.0196 | 1.1399 | 0.051 | 0.7411 | 0.0281 | 2.0379 | 0.3168 | $\hat{\alpha}_{Q,HillH,n}$
| $10000$ | 1.1529 | 0.0159 | 0.5177 | 0.0062 | 1.1374 | 0.0162 | 0.741 | 0.0089 | 2.0044 | 0.0932 | $\hat{\alpha}_{Q,HillH,n}$

The second group of two estimators

$\hat{\alpha}_{Frech,n}=-\frac{log(-log\,\widehat{p}_{eR})}{log\,\widehat{O}_{R}},\quad\hat{\alpha}_{Q,Frech,n}=-\frac{log(log(4))-log(log(4/3))}{log\widehat{Q}_{3}-log\widehat{Q}_{1}}$

is better in cases when the observed r.v. is a $Frechet(\alpha)$ distributed.
We should mention that in both of these cases, it is well known that for
estimating the parameter of the Pareto distribution, the Hill estimator (see
e.g. [Hill  _et al._(1975)Hill _et al._]) is the best estimator. With respect
to the robustness their behaviour is comparable with the one of the t-Hill
estimator (see [Jordanova and Pancheva(2012), Fabián and Stehlık(2009)]).

The last estimator is the most appropriate in case the observed r.v. has Hill-
Horror distribution

$F^{\leftarrow}(p)=(1-p)^{-1/\alpha}(-log\,(1-p)),p\in(0,1),\quad(see\cite[cite]{[\@@bibref{}{EMK}{}{}]}).$

This estimator is defined by

$\hat{\alpha}_{Q,HillH,n}=\frac{log(3)}{log\widehat{Q}_{3}+log(log(4/3))-log\widehat{Q}_{1}-log(log(4))}.$

Let us make a brief empirical investigation of these estimators. For different
but fixed $n=30,10^{2},10^{3},10^{4}$, we have made $m=10^{4}$ samples with
sample size $n$, of observation on one and the same r.v. Within these
$m=10^{4}$ samples the type and the parameters are one and the same, but in
general the types change between $Pareto(\alpha)$, $Frechet(\alpha)$ or $Hill-
Horror(\alpha)$ distribution for different $\alpha$. Then we have calculated
$10^{4}$ values of $\hat{\alpha}_{Par,n}$, $\hat{\alpha}_{Q,Par,n}$,
$\hat{\alpha}_{Frech,n}$, $\hat{\alpha}_{Q,Frech,n}$,
$\hat{\alpha}_{Q,HillH,n}$ and finally we have calculated the corresponding
means and standard deviations. The results are given in Table 5. The best
estimator in any particular case is the one that takes into account the type
of the observed r.v. Therefore the choice of the distribution is the most
important step for the estimation of the index of regular variation.

Although we have found good estimators for the regularly varying index when
the observed distribution is almost regularly varying. The most dangerous
case, is again the Hill-Horror distributed one. The question about estimation
of $\alpha$ for small samples of such data, e.g. when $n\leq 30$ is still
open. In this cases, however it seems to be not realistic to find a good
estimator of the tail index, because due to the slow rate of convergence, with
very high probability, the sample does not contain enough information about
the tail of the distribution.

##  4 ACKNOWLEDGMENTS

The authors was partially supported by the Project RD-08-96/06.02.2017 from
the Scientific Research Fund in University of Shumen and grant No
80-10-146/21.04.2017 of Sofia University, Bulgaria.

## References

  * [Irwin(1925)] J. Irwin, Biometrika 238–250 (1925).
  * [McKay(1935)] A. McKay, Biometrika 27, 466–471 (1935).
  * [Nair(1948)] K. Nair, Biometrika 35, 118–144 (1948).
  * [Dixon(1950)] W. J. Dixon, The Annals of Mathematical Statistics 21, 488–506 (1950).
  * [Dixon(1953)] W. Dixon, Biometrics 9, 74–89 (1953).
  * [Grubbs(1969)] F. E. Grubbs, Technometrics 11, 1–21 (1969).
  * [Klebanov(2016)] L. B. Klebanov, arXiv preprint arXiv:1611.05410 (2016).
  * [Klebanov  _et al._(2017)Klebanov, Antoch, Karlova, and Kakosyan] L. B. Klebanov, J. Antoch, A. Karlova, and A. V. Kakosyan, arXiv preprint arXiv:1701.06642 (2017).
  * [Klebanov, Kakosyan, and Karlova(2016)] L. B. Klebanov, A. V. Kakosyan, and A. Karlova, arXiv preprint arXiv:1612.09265 (2016).
  * [Tukey(1977)] J. W. Tukey, (1977).
  * [McGill, Tukey, and Larsen(1978)] R. McGill, J. W. Tukey, and W. A. Larsen, The American Statistician 32, 12–16 (1978).
  * [Parzen(1979)] E. Parzen, Journal of the American statistical association 74, 105–121 (1979).
  * [Hyndman and Fan(1996)] R. J. Hyndman and Y. Fan, The American Statistician 50, 361–365 (1996).
  * [Langford(2006)] E. Langford, Journal of Statistics Education 14, 1–27 (2006).
  * [Devore(2015)] J. L. Devore,  _Probability and Statistics for Engineering and the Sciences_ (Cengage Learning, 2015).
  * [SEMATHECH()] N. SEMATHECH, http://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm .
  * [Watkins, Scheaffer, and Cobb(2010)] A. E. Watkins, R. L. Scheaffer, and G. W. Cobb,  _Statistics: from data to decision_ (John Wiley & Sons, 2010).
  * [De Haan and Ferreira(2007)] L. De Haan and A. Ferreira,  _Extreme value theory: an introduction_ (Springer Sci. & Business Media, 2007).
  * [Hill  _et al._(1975)Hill _et al._] B. M. Hill _et al._ , The annals of statistics 3, 1163–1174 (1975).
  * [Jordanova and Pancheva(2012)] P. Jordanova and E. Pancheva, Comptes rendus de l’académie bulgare des sciences 65, 1649–1656 (2012).
  * [Fabián and Stehlık(2009)] Z. Fabián and M. Stehlık, (2009).
  * [Embrechts(1997)] K. C. M. T. Embrechts, P.,  _Modelling Extremal Events for Insurance and Finance._ (Springer, 1997).