# Do LLMs Dream of Ontologies?

Marco Bombieri1 Paolo Fiorini1 Simone Paolo Ponzetto2&Marco Rospocher1
1University of Verona, Verona, Italy
2University of Mannheim, Mannheim, Germany {marco.bombieri_01, paolo.fiorini,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) have recently revolutionized automated text
understanding and generation. The performance of these models relies on the
high number of parameters of the underlying neural architectures, which allows
LLMs to memorize part of the vast quantity of data seen during the training.
This paper investigates whether and to what extent general-purpose pre-trained
LLMs have memorized information from known ontologies. Our results show that
LLMs partially know ontologies: they can, and do indeed, memorize concepts
from ontologies mentioned in the text, but the level of memorization of their
concepts seems to vary proportionally to their popularity on the Web, the
primary source of their training material. We additionally propose new metrics
to estimate the degree of memorization of ontological information in LLMs by
measuring the consistency of the output produced across different prompt
repetitions, query languages, and degrees of determinism.

##  1 Introduction

Large Language Models (LLMs) have revolutionized how natural language is
computationally processed. Pre-trained models, such as GPT (Brown et al.,
2020), LLaMA (Touvron et al., 2023), and Bloom (BigScience Workshop, 2023)
have shown that human-level performance can be achieved on various tasks
(e.g., those found in SuperGLUE (Wang et al., 2019)) with no or minimal task-
specific adaptation.

Recent works (Chang et al., 2023; Carlini et al., 2023; Ranaldi et al., 2023,
_inter alia_) have investigated memorization aspects related to these models,
i.e., to what extent can the content used to train an LLM can be extracted as-
is through appropriate interaction with the model. These works have mainly
focused on extracting the actual sequences of tokens used to train the model.
Here, we investigate instead a different aspect of memorization, namely to
what extent general-purpose pre-trained LLMs can memorize information, rather
than specific token sequences. To this end, we assess whether and to what
extent some popular pre-trained general-purpose LLMs have memorized
information from known ontologies, focusing on a basic and fundamental piece
of information, namely the association of concept identifiers with their
corresponding natural language labels (e.g., GO:0001822 with kidney
development).

We first evaluate how much LLMs such as GPT and Pythia-12B (Biderman et al.,
2023) have acquired information about concepts from the text and can retrieve
it when prompted without further training. Our experiments, conducted on the
Gene (Ashburner et al., 2000) and Uberon (Mungall et al., 2012) ontologies,
show that, while LLMs show awareness of each of the ontologies we consider,
they have not memorized completely and consistently all concepts.
Consequently, we next investigate which ontological concepts have been better
memorized and why. Experimental findings indicate that the correct
memorization of concepts varies among them while correlating with the
frequency of their occurrence on the Web, thus suggesting that the information
was not acquired (or at least not exclusively) by processing the ontology
itself but rather from other textual Web materials mentioning it. Indeed, our
experiments advocate that the more information on some ontology concepts is
present in textual form on the Web, which is probably largely used as training
material for LLMs, the better they memorize it. Our findings relate with some
theories on the way the human memory works, such as the law of repetition:
indeed, while for humans, “the speed and accuracy of remembering improve with
repeated opportunities to study and retrieve the to-be-remembered material”
(Kahana et al., 2022), it seems that also for LLMs, memorizing information
about some ontology appears to depend on the number of times that information
has been seen in the training material. We finally focus on quantifying the
degree of memorization observed in previous experiments. For this, we ask for
the very same information multiple times to the LLM, using different prompt
repetitions, setting different temperature levels to control the model’s
output, and using different prompting languages. The experiments show that the
more the information is popular on the Web and thus likely to be more often
seen in the training material, the more the output of the model is consistent
and accurate independently of the prompt language and temperature level used,
thus suggesting that the invariance of the model output can be used as
evidence of information memorization. To foster further research on these
aspects, we make all code and data publicly available.

##  2 Related work

Several recent works have studied the memorization of different kinds of
information in language models for different purposes. Ranaldi et al. (2023)
investigate the interplay between memorization and performance in downstream
tasks with BERT. By defining a measure for evaluating memorization from pre-
training, they establish a correlation between highly memorized examples and
improved classification. Chang et al. (2023) explore memorization in ChatGPT
and GPT-4 through a ‘data archaeology’ approach. By employing a name cloze
membership inference query, they show that these models have memorized a
diverse range of copyrighted materials, with the extent of memorization linked
to the prevalence of passages on the web. Carlini et al. (2023) delve into the
undesirable consequences of memorization in LLMs. They demonstrate that LLMs,
when prompted appropriately, emit verbatim memorized training data, leading to
privacy violations, degraded utility, and fairness issues, a problem already
identified for clinical notes in BERT (Lehman et al., 2021) and personal
information in GPT-3 Huang et al. (2022). In response to security concerns
surrounding these models, Ishihara (2023) conducts a comprehensive survey on
training data extraction from Pre-trained Language Models (PLMs), providing a
taxonomy of memorization definitions and systemizing approaches for both
attack and defense. In our work, we follow this line of research while
focusing on _memorization of information_ rather than textual tokens and
sequences.

Recently, He et al. (2023) proposed to investigate PLMs’ knowledge of
ontologies using a set of inference-based probing tasks, thus showing that
these models encode little subsumption relations. Here, we complement their
work by focusing primarily on the terminology component and its provenance
with respect to the training data. The importance of our work stems from the
fact that PLMs are becoming ever more critical for ontology-centric tasks like
ontology mapping and alignment (Hertling and Paulheim, 2023), whose evaluation
may also suffer from dataset leakage (Sainz et al., 2023; Deng et al., 2023).

##  3 Do LLMs dream of ontologies?

###  3.1 Task description

To evaluate the memorization of ontological information in LLMs, we propose a
simple task inspired by that of exact memorization (Tirumala et al., 2022). In
particular, we ask the LLMs to return in a zero-shot fashion the ID of an
ontological concept, given in input, through a natural language prompt, the
concept’s label. No further training on domain data is thus performed, and no
information is inserted into the LLMs context other than that in the prompt.

We remark that the ID-concept label association is very basic information
contained in (many) ontologies, which typically include more complex and
structured knowledge – e.g., concept taxonomies, concept relations, axioms (He
et al., 2023): however, IDs and concept labels are (i) inherently sequences of
characters and (ii) frequently mentioned together in textual content (e.g.,
Webpages, scientific articles), thus making the ID-concept label association
something likely to be observed in the training material of LLMs and thus
helpful in evaluating memorization in LLMs.

###  3.2 Resources

#### Ontologies.

We opt for ontologies whose entities111In this work, use the terms entity and
concept interchangeably. are uniquely identified with an ID syntactically
unrelated to the entity’s label to follow a similar approach to that of exact
memorization analysis. Doing so guarantees that if the LLM associates the
correct ID to an input label, it is because the ID and the corresponding label
have been encountered in the training material and thus have been memorized,
as it is impossible to derive the ID from the label only. We focus on two
domain ontologies because domain-specific terms tend to have a lower polysemy
(Koeling et al., 2005) and thus allow to ignore issues related to label
disambiguation (i.e., cases where the same term can be associated with
multiple IDs).

We use the popular Gene Ontology (GO) (Ashburner et al., 2000). GO provides a
computational representation of current scientific knowledge about the
functions of protein and non-coding RNA molecules produced by genes from many
different organisms. It is widely used to support scientific research and is
thus popular in Web publications. It contains $42,854$ different concepts,
each having a label and uniquely identified by a numerical ID (a.k.a., GO ID).
For example, the label kidney development is associated with the concept whose
GO ID is 0001822, while the label kidney morphogenesis to GO ID 0060993. As
evident from the examples, without memorization, it would be impossible for
the model to derive the correct GO ID given only the label. The second
ontology analyzed is the Uberon Ontology (Mungall et al., 2012). Uberon is an
integrated cross-species anatomy ontology representing a variety of entities
classified according to anatomical structure, function, and developmental
lineage. In the Uberon ontology, there are $15,543$ terms (a.k.a., labels)
that are uniquely associated with a numerical UBERON ID that is specific to
the ontology, i.e., identified by the prefix "UBERON". The ontology also
incorporates terms from other ontologies (e.g., GO and BFO), which we have
excluded from the analysis as they are not native to Uberon. While the Uberon
ontology is more recent and less prevalent in the scientific literature than
GO222To quantify, on December 19, 2023, the GO’s paper (Ashburner et al.
(2000)) is cited on Google Scholar $38,553$ times, while the Uberon’s paper
(Mungall et al. (2012)) $686$ times., it allows us to compare performance
across different resources and subdomains, i.e., proteins vs. anatomy.

#### Large Language Models.

We consider three different LLMs. The first one is the 12 billion parameters
release of EleutherAI’s Pythia (Biderman et al., 2023) (named Pythia-12B now
on), trained on ThePile dataset (Gao et al., 2021). ThePile is a dataset
comprising 825 GiB of English text from different sources, including PubMed
Central (90.27 GiB of data) and PubMed Abstract (19.26 GiB), which are
biomedical portals and thus can include literature inherent to the Gene and
Uberon ontologies. We also tested two OpenAI chatbot
models333https://openai.com/chatgpt [Last access on December 19, 2023], i.e.,
GPT-3.5-Turbo (named GPT-3.5 now on) and GPT-4-0613 (called GPT-4 now on)
respectively. The first has about 175 billion parameters and uses a 16k
context window, while the second one has about 1 trillion parameters and uses
an 8k context window. The details on the training dataset of these two models
are not publicly available.

###  3.3 Research Questions

To study the degree of memorization of ontology IDs and labels in LLMs, we
address the following research questions:

RQ1

Can LLMs correctly predict the ID of the concepts in a known ontology, only
given the concepts’ label in the input prompt? Are there differences in
performance between the LLMs analyzed? Are there any common patterns in errors
made by LLMs?

RQ2

Does the accuracy of the ID prediction correlate with the number of times the
ID-concept label association is found on the Web and thus (likely) present and
frequent in the training material? Does the latter also influence errors made
by LLMs?

RQ3

Can we use the observed consistency of the ID prediction for a given prompted
label, under prompt repetitions or perturbations, as an indication of the
degree of memorization of that ID-concept label association?

The first two research questions investigate whether, how, and why LLMs have
memorized some basic ontological information. The third one looks more closely
into ways to empirically assess the degree of memorization in the model of
such ontological information by observing which extended the prediction of the
LLMs changes when prompting it multiple times for the same information.

##  4 Methods

###  4.1 LLMs memorisation of ontologies (RQ1)

We test the performance of the three considered LLMs on the prediction task
described in Section 3.1, for all concepts defined in each ontology presented
in Section 3.2, and compute the resulting accuracy, i.e., the number of times
the model exactly return the correct ID for the given concept’s label (e.g.,
with the label kidney development from GO, the answer is considered correct
only if the ID is precisely GO:0001822), over all the concepts in the
ontology.

Before the evaluation, we tested during development, both for GPT models and
Pythia-12B, different prompts on a sample of 100 random entities and selected
the ones that produced the highest accuracy on them (if multiple prompts
achieved similar performance in GPT models, the one with the least number of
input tokens was selected, to reduce the costs of the API queries). We chose
different prompts444All tested prompts are reported in the supplementary
material. for the various models to better cope with their diverse natures,
i.e., chatbots (GPTs) vs. general-purpose (Pythia-12B). Finally, we selected
the following prompts for GPT models:

  * •

Provide the GO ID for the label “l”. In the answer write only the
corresponding GO ID.

  * •

Provide the UBERON ID for the label “l”. In the answer write only the
corresponding UBERON ID.

while we selected the following prompts for the Pythia-12B model, where we
asked the model to complete the sentence by adding a total of a maximum of
$10$ new tokens:

  * •

In the Gene Ontology, the GO ID of the label “l” is GO:

  * •

In the Uberon Ontology, the Uberon ID of the label “l” is UBERON:

#### Quantifying error patterns.

To analyze possible common patterns in errors made by LLMs on the given
prediction task, i.e., predicting a sequence of digits (the concept ID) given
one or more word tokens (the concept label), we propose to investigate whether
some syntactic similarity (e.g., character or token-based) exists between the
gold and wrongly-predicted IDs, or the corresponding concept labels, following
two complementary strategies.

The first approach investigates if the model tends to make mistakes by
providing an ID whose corresponding label is similar to the correct one. For
example, the label heart valve morphogenesis (GO ID GO:0003179) is close to
heart trabecula morphogenesis (GO ID GO:0061384) due to two words in common,
or the label regulation of cell division (GO ID GO:0051302) with regulation of
cell motility  (GO ID GO:2000145) because of three words in common. In these
examples, the numerical IDs of the two concepts are very different, but the
label has sub-tokens in common, which could lead to an error in the model. To
do this analysis, we extract the list of errors committed by the models by
collecting the gold label (i.e., the correct one) and the label corresponding
to the wrong ID predicted by the model. Then, we divide the two strings into
tokens and compute the Jaccard Similarity (Murphy, 1996) on them.

The second approach investigates if the model tends to make mistakes by
providing an ID similar to the correct one. For example, the GO ID GO:0060219
(camera-type eye photoreceptor cell differentiation) is close to GO:0060519
(cell adhesion involved in prostatic bud elongation) because only one digit
needs to be changed to go from the first to the second. To do this analysis,
we extract the list of wrong IDs predicted by the models and the gold one, and
we compute the Levensthein Distance (Levenshtein, 1966) on them.

Furthermore, we also check how many different IDs are predicted by the models
to understand to what extent they cover the whole ontologies with their
predictions.

###  4.2 Correlation between memorisation and popularity on the Web (RQ2)

RQ2 aims to asses the correlation (if any) between the popularity of a concept
on the Web, and thus likely in the LLM’s training material, and the model’s
memorization of it. We recall that precise information on the actual training
material used to build some models (namely, GPT-3.5 and GPT-4) is lacking.
However, it is pretty established that all of them were trained (at least) on
a massive amount of Web content (Chang et al., 2023). We thus construct a
dataset to investigate this relationship, containing information on the
popularity of the $(ID,label)$ pair on the Web, approximated as the number of
times the pair appears in the same documents on the Web. To estimate the
number of documents on the Web containing the pair $(ID,label)$, we use Google
Search APIs, with the query “label” “ID” for an exact match in the Web
documents. Despite the implementation details of Google Search are not
publicly released, and thus, it has to be used as a black box (Kilgarriff,
2007), it is commonly used for similar purposes (Chang et al., 2023).

For our analysis, we divide the ontology concepts into buckets according to
the $(ID,label)$ popularity. In particular, we calculate $50$ percentiles on
the set containing the number of Web occurrences, used as boundaries to divide
the ontology elements into $50$ buckets. Given the distribution of
occurrences, the first buckets will contain more elements than the last ones.
We set the value to $50$ to ensure that the buckets obtained have a minimum
number of elements, which we arbitrarily set to $35$. Figure 1 reports the
mean number of elements per bucket (on the left) and the average number of Web
occurrences per bucket (on the right).

Figure 1: Mean number of elements per bucket and average number of web
occurrences per bucket.

We then analyze the variation of the average accuracy of the model’s ID
prediction per bucket (i.e., the proportion of correctly predicted IDs for the
concepts in the buckets) with respect to the popularity on the Web of the
concepts in the bucket, assessing a possible correlation via Spearman’s rank.

#### Error patterns vs. popularity.

We also complement the study of possible common patterns in mistakes made by
LLMs on the given prediction task (cf. Section 4.1) with further analysis
related to the popularity of the concepts. First, we check whether the
similarity between wrong and gold predictions based on Levensthein distance
and Jaccard similarity correlates with the number of Web occurrences of the
concepts. Then, we investigate if the model is biased towards popular
concepts, i.e., if it tends to answer with the ID of a very popular concept in
case of uncertainty. For the latter, we extract the most common incorrectly
predicted IDs and check which buckets they belong to estimate if there is a
statistical correlation (via Spearman’s rank) between frequently predicted IDs
and their bucket number.

###  4.3 Assessing the degree of memorization (RQ3)

To assess the degree of memorization of the label-ID association of an
ontological concept (cf., RQ3), we propose to use the invariance of the
model’s response to the prompt as a metric: that is, we suggest to use the
variability of the answer of the model when perturbing the prompt differently
as a metric to estimate the degree of memorization. In particular, starting
from the prompt selected in Section 4.1, we test three different
implementations of the prediction invariance metric listed below:

  * •

PI-1: we repeat the same prompt more times with the temperature set to zero,
i.e., asking for determinism in the answer and counting the number of
different answers. Intuitively, if the concept is well memorized in the model,
only one answer should be returned.

  * •

PI-2: we repeat the same prompt with different temperature levels, i.e., from
$0.0$ to $1.0$ with $0.1$ increments. Intuitively, if the concept is well
memorized in the model, the answer should be independent of the temperature,
i.e., it should be the same, independently from the creativity of the model.

  * •

PI-3: we repeat the same prompt in different languages (English, Italian,
German, French, and Spanish) with the temperature set to zero (i.e., asking
for determinism). In this implementation, we translate only the prompt while
we keep the ontology’s labels in the original language, i.e., English.

We consider $20$ random concepts per bucket,555While the approach could be
applied to all concepts of an ontology, we limit to $20$ to economize on the
costs of the LLM APIs. for a total of $1000$ instances, allowing us to study
the impact of the three invariance strategies on the model’s behavior
according to the popularity of the concepts.

Given a set of $N$ buckets $B_{i}$, with $i\in[1,2,....,N]$ ($N=50$ in our
setting), each containing $K$ random concepts $C_{Bi}=[c_{1},c_{2},...,c_{K}]$
($K=20$ in our setting), we evaluate the performance by considering the mean
number $U$ of unique answers returned by the model for concepts belonging to
each bucket $B_{i}$. In particular, for each concept $c_{j}\in Bi$, we compute
the Prediction Invariance after $M$ prompt repetitions as
$PI_{c_{j}}^{M}=1-(\frac{U-1}{M-1})$, where $U$ is the number of unique
answers returned after $M$ prompts. If the $M$ prompts return always the same
answer, $PI_{c_{j}}$ will be $1.0$ (i.e., max prediction invariance). If the
$M$ prompts return $M$ different answers, $PI_{c_{j}}$ will be $0.0$ (i.e.,
minimum prediction invariance). In PI-1, $M$ is set to $10$, i.e., we repeat
as-is the same prompt $10$ times with the temperature set to zero. In PI-2,
$M$ equals the number of different temperature levels tested, i.e., $11$. In
PI-3, $M$ is the number of different languages tested, i.e., $5$. Finally, we
average the different $PI_{c_{j}}$ obtained on concepts $c_{j}\in B_{i}$,
obtaining the Average Prediction Invariance for each bucket $B_{i}$ as
$AvPI_{(B_{i})}=\frac{1}{K}\sum_{c_{j}\in C_{B_{i}}}PI_{c_{j}}^{M}$. We use
Spearman’s rank correlation coefficient to study the relationship between
$AvPI_{(B_{i})}$ and accuracy for the same bucket $B_{i}$ (i.e., the
correctness of the predictions).

##  5 Results and discussions

###  5.1 LLMs memorisation of ontologies (RQ1)

Table 1 reports the accuracy of the three compared models on the task
described in Section 3.1, for the two considered ontologies.

Table 1: Accuracy of Pythia-12B, GPT-3.5 and GPT-4 on the task described in Section 3.1, on the GO and Uberon datasets Dataset | Pythia-12B | GPT-3.5 | GPT-4
---|---|---|---
Gene Ontology | .007 | .061 | .127
Uberon Ontology | .000 | .003 | .013

Overall, the accuracy scores are quite low for all models and ontologies
considered.

On the Gene Ontology, GPT-4 has an accuracy of $.127$, almost doubling the
score of GPT-3.5 ($.061$): this result is in line with the findings of the
other papers on LLMs tokens memorization, e.g., Carlini et al. (2023).
Pythia-12B achieves the lower accuracy score at $.007$. The same ranking, but
with even lower accuracy scores, can also be observed on the Uberon Ontology:
GPT-4 achieves the highest accuracy score ($.013$), followed by GPT-3.5
($.003$) and then Pythia-12B ($.000$). We can observe that:

  * •

for both datasets, GPT-4 is the best-performing model, followed by GPT-3.5,
with Pythia-12B achieving the lowest scores: a possible explanation is the
different number of model parameters (c.f. Section 3.2) or the different
amount of data used in the training phase (we recall that, while for
Pythia-12B the dataset used for training is declared, for GPT models it is
not);

  * •

the performances on the Uberon Ontology are lower than on the Gene Ontology,
for all the considered models: a possible explanation is the lesser
popularity, as observed also on the Web, of the former compared to the latter
(cf., Section 3.2);

  * •

the better performance of both GPT models compared to Pythia-12B seems to
suggest that the former are trained on much more content than the latter: for
instance, the zero accuracy of Pythia-12B on Uberon can be ascribed to the
almost total absence of Uberon IDs in its training dataset ($15,494$ of the
$15,543$ Uberon IDs are not present at all in ThePile), which instead contains
occurrences of several Gene Ontology IDs.

#### Error patterns.

Looking more closely at the errors made by LLMs on the task, Table 2
summarizes the resulting Jaccard Similarity and Levensthein distance, averaged
over all the wrong predictions by the models on both the Gene and Uberon
ontologies.

Table 2: Average similarity measures on gold and predicted IDs (Jaccard similarity, abbreviated with Jaccard s.) and associated concept labels (Levensthein distance, abbreviated with Levensthein d.) on Gene Ontology (GO) and Uberon Ontology (UO). Measure | Pythia-12B | GPT-3.5 | GPT-4
---|---|---|---
Jaccard s. (GO) | .149 | .301 | .338
Levensthein d. (GO) | 4.409 | 3.964 | 3.814
Jaccard s. (UO) | .001 | .047 | .033
Levensthein d. (UO) | 6.570 | 3.533 | 4.580

In the Gene Ontology, GPT-4 scores the highest Jaccard Similarity, followed by
GPT-3.5 and Pythia-12B, thus showing that it predicts (wrong) IDs whose
associated concept labels are more similar to the one corresponding to gold
IDs than the other models. Analogously, GPT-4 scores the lowest Levensthein
distance, followed by GPT-3.5 and Pythia-12B, thus revealing that it predicts
(wrong) IDs closer to the gold one than the other two models. In the Uberon
Ontology, it is instead GPT-3.5 to obtain the highest Jaccard Similarity score
and lowest Levensthein distance, followed by GPT-4 and Pythia-12B. The values
obtained in this ontology are, however, much worse than those obtained in the
Gene Ontology, a fact that can be explained by the less deep knowledge of the
Uberon ontology by the LLMs as reported in Table 1.

During the error analysis, we also observed that each model repeatedly
predicts the same (wrong) IDs for different concept labels. Indeed,
considering the set of the predicted IDs by each model for all concept labels
in the ontology, we notice that only a fraction of all IDs were actually
emitted. Table 3 summarizes the number of unique IDs produced by the models
for both considered datasets, also indicating the percentage concerning the
number of concepts in the ontologies (c.f., Section 3.2).

Table 3: Number of unique predicted IDs and percentage wrt. all the IDs in the corresponding ontology. | Pythia-12B | GPT-3.5 | GPT-4
---|---|---|---
GO | 1353 ($3.16\%$) | 7182 ($16.76\%$) | 12308 ($28.72\%$)
UO | 30 ($0.19\%$) | 641 ($4.12\%$) | 2971 ($19.11\%$)

All models missed to produce a substantial part of the IDs of each ontology
(from $71.28\%$ to $96.84\%$ for the Gene Ontology, from $80.89\%$ to
$99.81\%$ for Uberon). GPT-4 predicted substantially more unique IDs than the
other two models, a result in line with its overall better accuracy (cf.,
Table 1).

Overall, to answer RQ1, the results show that the considered LLMs have minimal
knowledge of the information on the considered ontologies, that GPT-4
outperforms all the other analyzed models, and that some patterns can be found
in wrong predictions.

###  5.2 Correlation between memorisation and popularity on the Web (RQ2)

The two plots in Figure 2 show how the accuracy of the ID predictions of the
models, based on the prompted concept label from the Gene Ontology,666Given
the very low performance on the Uberon ontology for all models – cf. Section
5.1, we consider only the Gene Ontology to address this research question.
varies according to how frequently that concept occurs on the Web. We recall
(c.f. Section 3.2) that concepts were organized in different buckets, labeled
from B1 (least frequent concepts - basically not observed) to B50 (most
frequent concepts - observed tens of thousands of times) based on the number
of occurrences of the concepts in the Web. Both plots report the average
prediction accuracy of the considered models, but using different variables on
the x-axis: the average concept’s Web occurrences per bucket in Figure 2(a),
while instead the ordinal number of the bucket in Figure 2(b).

(a)

(b)

Figure 2:  Variation of the average accuracy of the model’s ID prediction
according to the popularity of the concept on the Web: (a) plot of the average
accuracy per bucket with respect to the average concept’s occurrences per
bucket on the x-axis; (b) plot of the same average accuracy per bucket but
with respect to the ordinal number of the corresponding bucket on the x-axis.

First of all, as expected by the results discussed in Section 5.1, the
accuracy score of GPT-4 on each bucket is higher than that of GPT-3.5, which
in turn is higher than that of Pythia-12B.

Furthermore, both plots show that the average accuracy per bucket of the
predictions increases the more the concepts are observed on the Web for all
three considered models. That is, the plots show that the considered models
tend to make correct predictions for concepts frequently observed on the Web
(and, thus, likely commonly observed in the training material), and wrong
predictions for less frequently observed concepts, thus suggesting a certain
degree of correlation between the accuracy of the predictions (averaged per
bucket), i.e., the degree of concepts correct memorization, and the occurrence
of the concepts in the Web. To further confirm this, we computed the
Spearman’s rank correlation coefficient between the ranking of the buckets
according to the averaged prediction accuracy per bucket and the ranking of
the buckets according to the average number of Web occurrences of the concepts
in the bucket, obtaining strong correlation for all three models – GPT-4:
$0.98$; GPT-3.5: $0.99$; Pythia-12B: $0.85$ – values statistically significant
according to the Permutation test.

#### Error patterns vs popularity.

We also investigated if any correlation exists between the average Jaccard
similarity / Levensthein distance computed for the single buckets of the
dataset and the average number of Web occurrences of the concepts in them.
Table 4 reports the Spearman’s rank correlation coefficient between the
ranking of the buckets according to the averaged Jaccard similarity /
Levensthein distance per bucket and the ranking of the buckets according to
the average number of Web occurrences of the concepts in the bucket.

Table 4: Spearman’s rank correlation coefficient between the number of Web occurrences of the concepts and the considered similarity measures. The “*” means the correlation is statistically significant (p-value $\leq.05$). Similarity Measure | Pythia-12B | GPT-3.5 | GPT-4
---|---|---|---
Jaccard similarity | .788* | .590* | .595*
Levensthein distance | -.714* | -.364* | -.208

The results show a moderate (GPT-3.5) and strong correlation (Pythia-12B,
GPT-4) with the Jaccard Similarity (statistically significant according to the
Permutation test in all cases), while a strong (Pythia-12B) and weak (GPT-3.5,
GPT-4) inverse correlation (statistically significant according to the
Permutation test in two cases out of three) with the Levensthein distance.
Overall, the results suggest that when the models make prediction errors, the
wrongly predicted IDs are syntactically closer to the gold one (directly or
indirectly through the associated labels) for concepts frequently occurring on
the Web than infrequent ones. Because of our underlying assumption on the
training material and the previous findings related to RQ2, this suggests
further that the more a concept is seen in the training data, the more likely
the prediction is either correct or more closely related to the gold one.
Overall, the observed results suggest a positive answer to RQ2.

We further investigated the repeated IDs phenomenon discussed at the end of
Section 5.1 in light of the popularity data. We collected the top-500 most
repeated IDs for each model for the Gene Ontology and checked to which bucket
they belong. More in detail, for each bucket, we computed the ratio between
how many top-500 repeated IDs belong to it and how many of them would be
expected to belong to it, assuming the top-500 IDs are proportionally
allocated to each bucket according to the actual distribution of the concepts
in the buckets (Figure 1). The plot in Figure 3 shows the ratio between actual
and expected top-500 repeated IDs per bucket.

Figure 3: “Proportional” distribution of the top-500 repeated IDs in the
buckets of the dataset.

The plot clearly shows that for Pythia-12B most of the top-500 repeated IDs
proportionally belong to the buckets containing the most occurring concepts on
the Web. A similar yet less prominent trend can also be observed for GPT-3.5,
while for GPT-4 the top-500 repeated IDs seem to be more proportionally
distributed among all the buckets. That is, Pythia-12B and GPT-3.5 seem to
repeatedly predict IDs more often for popular Web concepts, conversely than
what happens for GPT-4. Relying again on Spearman’s rank correlation
coefficient to measure a possible correlation between the considered
variables, a strong correlation can be observed for Pythia-12B (.868) and
GPT-3.5 (.947), both statistically significant according to the Permutation
test, while no correlation can be observed for GPT-4.

###  5.3 Assessing the degree of memorization (RQ3)

We finally investigated if the prediction invariance methods proposed in
Section 4.3 can be used to estimate the degree of memorization of the ID-
concept label association in LLMs. We applied the three proposed methods
(PI-1: as-is repetition, PI-2: temperature variation, PI-3: language
variation) with GPT-3.5 and the Gene Ontology dataset. Figures 4(a) and 4(b)
show the trend of the $AvPI$ and the accuracy, respectively, calculated on a
sample of $20$ random instances per bucket.

For frequently occurring concepts, i.e., those belonging to the very last
buckets, all the methods achieve an average prediction invariance (AvPI) score
close to 1; that is, the model almost always predicts the same ID for a given
concept label, and the prediction is almost always the correct one (accuracy
close to 1 as well). For infrequently occurring concepts, i.e., those
belonging to the first buckets, while PI-1 achieves an AvPI score close to 1,
the application of PI-2 and PI-3 results in a higher variability of the model
predictions, i.e., lower AvPI score, which is more substantial when the prompt
is repeated with different temperature levels (PI-2). Interestingly, in bucket
B1, all methods return an average accuracy of zero, meaning that even if the
methods tried different ID predictions among the various repetitions, none of
them was the correct one.

The plot in Figure 4(a) has an increasing trend for both PI-2 and PI-3,
indicating that the more a concept is repeated on the Web (and therefore
potentially in the model’s training material), the more invariant the model
predictions are when solicited with PI-2 and PI-3 methods. If this information
is compared with the trend of the graph in Figure 4(b) and with the fact that
the Spearman’s rank coefficient between AvPI and accuracy is high (very strong
correlation) for both PI-2 and PI-3 ($.95$ and $.85$ respectively) and
statistically significant according to Permutation test, we conclude that both
these methods may be suitable for assessing the memorization of exact
information in LLMs. On the contrary, the PI-1 method is unable to
discriminate memorized information from non-memorized one.

(a) AvPI per bucket.

(b) Accuracy per bucket.

Figure 4:  AvPI (Subfigure 4(a)) and corresponding accuracy (Figure 4(b)) on
$20$ samples per bucket after applying PI-1, PI-2, and PI-3.

While a more comprehensive analysis is necessary, these preliminary findings
seem to suggest that looking at the variation of the predictions of an LLM
(such as GPT-3.5 in these experiments) when invoked multiple times with the
same prompt but varying temperature level (PI-2) or language (PI-3) may
provide hints on the degree of memorization of that information in the model,
an thus how likely it is not a result of some model hallucination: if the
model (almost) always predicts the same information (AvPI close to 1), it is
potentially because it has seen that information many times in the training
material, and thus has memorized it. Furthermore, this prompt-invariant answer
is likely correct, as suggested by the positive correlation between AvPI and
accuracy. Conversely, suppose the model tends to predict different information
(AvPI heading toward 0) by varying the prompt. In that case, it may be because
the LLM has not memorized that concept due to its scarce occurrence in the
pretraining data.

##  6 Conclusions

This paper investigated whether LLMs have correctly memorized the label-ID
association in ontologies, finding that only a small fraction of concepts is
memorized. Among the LLMs analyzed, GPT-4 is the one that obtains the highest
performance. The paper, therefore, investigated why some instances are
correctly memorized and others not, finding for all the LLMs a strong
correlation between the popularity of the concept (i.e., the number of concept
occurrences on the Web) and the correctness of the prediction. This
correlation may be ascribed to the fact that the resources used for training
the models likely include a substantial amount of textual material coming from
the Web. Furthermore, the conducted error analyses show that the more a
concept is popular and likely seen in the training data of LLMs, the more the
wrong prediction is anyway closely related, in terms of Jaccard similarity, to
the gold one. Moreover, this paper proposed the Prediction Invariance metric
and evaluated different strategies for assessing the degree of correct
memorization of concepts in LLMs, showing that the invariance of the model
output to the language of the prompt or the model temperature can be used as
evidence of information memorization. In future work, we will extend this
analysis to the memorization of other ontological information.

## Ethical Statement

There are no ethical issues.

## References

  * Ashburner et al. [2000] M Ashburner, C A Ball, J A Blake, D Botstein, H Butler, J M Cherry, A P Davis, K Dolinski, S S Dwight, J T Eppig, M A Harris, D P Hill, L Issel-Tarver, A Kasarskis, S Lewis, J C Matese, J E Richardson, M Ringwald, G M Rubin, and G Sherlock.  Gene ontology: tool for the unification of biology. the gene ontology consortium.  Nat Genet, 25(1):25–29, May 2000.
  * Biderman et al. [2023] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal.  Pythia: A suite for analyzing large language models across training and scaling.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023.
  * BigScience Workshop [2023] BigScience Workshop.  Bloom: A 176b-parameter open-access multilingual language model.  CoRR, abs/2211.05100, 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * Carlini et al. [2023] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang.  Quantifying memorization across neural language models.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  * Chang et al. [2023] Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman.  Speak, memory: An archaeology of books known to chatgpt/gpt-4.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 7312–7327. Association for Computational Linguistics, 2023.
  * Deng et al. [2023] Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan.  Investigating data contamination in modern benchmarks for large language models, 2023.
  * Gao et al. [2021] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The pile: An 800gb dataset of diverse text for language modeling.  CoRR, abs/2101.00027, 2021.
  * He et al. [2023] Yuan He, Jiaoyan Chen, Ernesto Jiménez-Ruiz, Hang Dong, and Ian Horrocks.  Language model analysis for ontology subsumption inference.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 3439–3453. Association for Computational Linguistics, 2023.
  * Hertling and Paulheim [2023] Sven Hertling and Heiko Paulheim.  OLaLa: Ontology matching with large language models.  CoRR, abs/2311.03837, 2023.
  * Huang et al. [2022] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang.  Are large pre-trained language models leaking your personal information?  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038–2047, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * Ishihara [2023] Shotaro Ishihara.  Training data extraction from pre-trained language models: A survey.  CoRR, abs/2305.16157, 2023.
  * Kahana et al. [2022] Michael J Kahana, Nicholas B Diamond, and Ada Aka.  Laws of human memory, Apr 2022.
  * Kilgarriff [2007] Adam Kilgarriff.  Last words: Googleology is bad science.  Computational Linguistics, 33(1):147–151, 2007.
  * Koeling et al. [2005] Rob Koeling, Diana McCarthy, and John Carroll.  Domain-specific sense distributions and predominant sense acquisition.  In Raymond Mooney, Chris Brew, Lee-Feng Chien, and Katrin Kirchhoff, editors, Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 419–426, Vancouver, British Columbia, Canada, October 2005. Association for Computational Linguistics.
  * Lehman et al. [2021] Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace.  Does BERT pretrained on clinical notes reveal sensitive data?  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 946–959, Online, June 2021. Association for Computational Linguistics.
  * Levenshtein [1966] Vladimir Iosifovich Levenshtein.  Binary codes capable of correcting deletions, insertions and reversals.  Soviet Physics Doklady, 10(8):707–710, feb 1966.  Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.
  * Mungall et al. [2012] Christopher J. Mungall, Carlo Torniai, Georgios V. Gkoutos, Suzanna E. Lewis, and Melissa A. Haendel.  Uberon, an integrative multi-species anatomy ontology.  GenomeBiology.com, 13(1), 1 2012.
  * Murphy [1996] Allan H. Murphy.  The finley affair: A signal event in the history of forecast verification.  Weather and Forecasting, 11(1):3 – 20, 1996.
  * Ranaldi et al. [2023] Leonardo Ranaldi, Elena Sofia Ruzzetti, and Fabio Massimo Zanzotto.  PreCog: Exploring the relation between memorization and performance in pre-trained language models.  In Ruslan Mitkov and Galia Angelova, editors, Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, RANLP 2023, Varna, Bulgaria, 4-6 September 2023, pages 961–967. INCOMA Ltd., Shoumen, Bulgaria, 2023.
  * Sainz et al. [2023] Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre.  NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore, December 2023. Association for Computational Linguistics.
  * Tirumala et al. [2022] Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan.  Memorization without overfitting: Analyzing the training dynamics of large language models.  In NeurIPS, 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Wang et al. [2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.