# Analogy Generation by Prompting Large Language Models: A Case Study of
InstructGPT

Bhavya Bhavya  {bhavya2<EMAIL_ADDRESS>Jinjun Xiong<EMAIL_ADDRESS>ChengXiang Zhai  {bhavya2<EMAIL_ADDRESS>
###### Abstract

We propose a novel application of prompting Pre-trained Language Models (PLMs)
to generate analogies and study how to design effective prompts for two task
settings: generating a source concept analogous to a given target concept (aka
Analogous Concept Generation or acg), and generating an explanation of the
similarity between a given pair of target concept and source concept (aka
Analogous Explanation Generation or aeg). We found that it is feasible to
prompt InstructGPT to generate meaningful analogies and the best prompts tend
to be precise imperative statements especially with a low temperature setting.
We also systematically analyzed the sensitivity of the InstructGPT model to
prompt design, temperature, and injected spelling errors, and found that the
model is particularly sensitive to certain variations (e.g., questions vs.
imperative statements). Further, we conducted human evaluation on 1.4k of the
generated analogies and found that the quality of generations varies
substantially by model size. The largest InstructGPT model can achieve human-
level performance at generating meaningful analogies for a given target while
there is still room for improvement on the aeg task.111Our code and datasets
are available for public use: https://github.com/Bhaavya/InstructGPT-Analogies

##  1 Introduction

Table 1: Selected prompts and InstructGPT-generated analogies for natural
selection

Prompt (P7): |  What is analogous to natural selection?
---|---
InstructGPT Output: |  The analogous process to natural selection is artificial selection. (9 words)
Prompt (P2): |  Explain natural selection using a well-known analogy.
InstructGPT Output: |  Imagine that you have a jar of mixed nuts … If you shake the jar …the big nuts will fall out first … analogy is that natural selection is like a sieve that separates the fit from the unfit… (136 words)

.

Large Pre-trained Language Models (PLMs) such as BERTDevlin et al. (2018) and
GPTBrown et al. (2020) have been applied to many tasks of text generation
(e.g., summarization, dialogue system) with promising results Li et al.
(2021). However, no existing work has studied how to apply PLMs to generate
different kinds of textual analogies, such as conceptual metaphors (e.g.,“Life
is a journey222https://en.wikipedia.org/wiki/Conceptual_metaphor”), and
instructional analogies (e.g., “A red blood cell is like a truck in that they
both transport essential supplies”Newby et al. (1995)).

Generating analogies has a wide range of applications, such as explaining
concepts and scientific innovation, and analogies play a crucial role in human
cognition. Analogical matching and reasoning enables humans to understand and
learn unfamiliar concepts (aka target concepts) by means of familiar ones (aka
source concepts) and to make scientific innovations. Unsurprisingly, analogy
modeling and generation has been a long-standing goal of AI Mitchell (2021).
This is a challenging problem because it often requires computing deep
semantic similarities that are beyond the surface-level similarity. For
example, the Bohr’s atom model and the solar system are analogous due to their
structural and relational similarities (i.e., atoms orbit around the nucleus
like planets around the sun).

Much work has been done to compute such analogical similarities between
concepts. However, existing approaches mostly rely on structured
representations, thus, they can only where such representations already exist.
For example, one of the most popular models is Structural Mapping Engine (SME)
Forbus et al. (2017), which aligns structured representations of the target
and source concepts using predicate logic. Moreover, they cannot generate
analogies in natural language.

Inspired by the recent success in applying PLMs to many NLP tasks (e.g., Li et
al. (2021)), we propose and study the application of PLMs to analogy
generation. We consider two typical application scenarios of analogy
generation: 1) Analogous Concept Generation (acg): given a target concept
(e.,g, bohr’s model), generate a source concept analogous to the target
concept (e.g., solar system), possibly with an explanation of their
similarities; 2) Analogy Explanation Generation (aeg): given a target concept
and an analogous source concept, generate an explanation of their
similarities.

By noting the similarity of the two tasks defined above to other text
generation problems, and being inspired by the recent success of using
prompted PLMs for text generation, we propose analogy generation by using a
PLM with appropriately designed prompts. We adopt the promising emerging
paradigm of prompting language models Liu et al. (2021) that uses textual
prompts with unfilled slots and directly leverages the language models to fill
those slots and obtain the desired output. For example, Table
LABEL:tab_sample_std shows sample prompts and PLM-generated outputs for acg
from our experiments.

Specifically, we study the following main research questions: RQ1) How
effective is a modern PLM such as InstructGPT in generating meaningful
analogies? RQ2) How sensitive are the generated analogies to prompt design,
the temperature hyperparameter, and spelling errors? RQ3) How does the model
size impact the quality of generated analogies?

To study these questions, we design several experiments on analogies generated
from the InstructGPT Ouyang et al. (2022) model. First, we manually validate
whether InstructGPT can generate meaningful analogies for ten well-known
analogies in the science domain. Next, we design and systematically vary
prompt variants (e.g., imperative statements vs. questions) and temperature,
and investigate the corresponding variations in the generated text by
comparing them to a reference dataset of science analogies. Finally, we study
the impact of model size on the quality of generated analogies both by
automatically comparing against the reference data and using human evaluation.

Our experimental results show that PLMs (specifically, InstructGPT) offer a
promising general approach to generating analogies with properly designed
prompts. Furthermore, the InstructGPT model is found to be sensitive to the
prompt design, temperature, and spelling errors for this task, particularly to
the prompt style (i.e., question vs. imperative statement). Precise imperative
statements in low-temperature setting are found to be the best prompts.
Finally, the quality of the generated analogies depends heavily on the model
size. While the largest model can achieve human-level performance on the acg
task, the smallest model barely generates any meaningful analogies. The aeg
task proved to be more challenging based on human evaluation and could be a
better test of the analogical reasoning capabilities of PLMs especially for
explaining analogies not seen during training.

##  2 Related Work

###  2.1 Computational Models of Analogies

There has been a lot of work on computational modeling of analogies Mitchell
(2021). The SME model Forbus et al. (2017) is one of the most popular symbolic
model that finds the mapping, or connections between structured
representations of source and target concepts and their attributes. However,
such methods cannot generate new analogous source concepts with analogical
explanation.

The recent deep learning-based approaches, including using pre-trained
language models Mikolov et al. (2013); Rossiello et al. (2019); Ushio et al.
(2021), are able to generate analogies to some extent, but are currently
limited to simple word-level and proportional analogies, such as (ostrich:bird
:: lion:?). In contrast, we aim to generate and explain more complex analogies
of concepts, e.g. instructional analogies Newby et al. (1995).

Another line of work is on finding analogous documents for scientific
innovation, such as product descriptions and research papers, based on their
semantic similarities Kittur et al. (2019). In contrast, we operate in a
generative task setup.

To the best of our knowledge, none of the existing work has studied the
problem of automatically generating complex analogies in natural language.
Recently, research on more “generative” analogy-making tasks has been
recommended Mitchell (2021). Along this direction, we believe that our
proposed task is challenging and more practically useful than the existing
text-based generative analogical tasks including letter-string (e.g., if “abc”
changes “abd”), what does “pqrs” change to?) and word-level analogies.

.

###  2.2 Prompting Language Models

Recently, prompts have been either manually created or learned to successfully
leverage PLMs for several natural language tasks Liu et al. (2021). Our work
is closest to prompting for lexical and proportional analogy generation Ushio
et al. (2021). But, none of the existing work has performed an in-depth study
on prompting PLMs for both generating analogous concepts given a single query
concept and explaining the analogical similarities between two query concepts.

##  3 Problem Formulation

Motivated by the practical applications of this task (e.g., explaining
concepts), we study analogy generation in the following settings.

1\. Analogous Concept Generation (acg) or No Source (no_src): Here, only the
target concept is provided as the input. The goal is to generate an analogous
source concept or scenario, along with some explanation to justify the
analogy. For example, “Explain Bohr’s atomic model using an analogy.”

2\. Analogy Explanation Generation (aeg) or With Source (wsrc): Here, in
addition to the target, the source concept is also a part of input. The goal
is to generate an explanation of how the target and source are analogous. For
example, “Explain how Bohr’s atomic model is analogous to the solar system.”

Our problem setup is similar to the use of PLMs for text generation Li et al.
(2021), and is most closely related to single-relation analogy generation
(e.g., ostrich : bird :: animal : lion) Ushio et al. (2021), where the input
is a pair of query concept (e.g., ostrich : bird), and the task is to choose
an analogical pair from a pre-defined list of candidate pairs. But, our
proposed task is still different in nature and much more challenging (e.g.,
requiring more creativity in some cases). First, both of our inputs and
outputs are different. For example, in the proposed acg setup, our input is a
single concept (e.g., “bohr’s model”), not a pair of concepts. Our task is to
identify another concept (or scenario) that has an equivalence to the query
concept based on their deep and non-trivial semantic similarities. No previous
work has studied this kind of “single-concept-based” analogy generation with
pre-trained language models. Even in the proposed aeg setup where we also use
a pair of concepts as input, they are different from the pair used in the
previous work. For example, our input could be a pair (e.g., “bohr’s model”
and “solar system”) and the output is an explanation of their analogical
relations (e.g., how their structures are similar). Second, we do not have a
pre-defined finite list of candidates to choose from, which is a more
realistic and interesting setting than previous work from application
perspectives, and is also much more challenging for evaluation.

##  4 Experiment Setup

In this section, we discuss InstructGPT PLM and datasets used in our
experiments.

InstructGPT Model: Recently, several PLMs have been developed and trained on
massive web data Devlin et al. (2018); Brown et al. (2020); Raffel et al.
(2019). In this study, we probe the aligned GPT-3 models, InstructGPT. These
are GPT-3 models that have been optimized to follow instructions better Ouyang
et al. (2022). InstructGPT has four variants depending on the model size
(number of parameters), namely Ada (350 M), Babbage (1.3 B), Curie (6.7 B) ,
and Davinci (175 B)333https://blog.eleuther.ai/gpt3-model-sizes/. Unless
otherwise mentioned, we use the Davinci model for the experiments as it is
expected to have the best performance.

We used the Open AI API 444https://beta.openai.com/docs/api-
reference/completions/create to generate all analogies. Main hyperparameters
are described in Section 5.2.2 and rest in the Appendix A

Dataset: As the task of analogy generation, as defined in this paper, has not
been previously studied, there is no existing data set available to use
directly for evaluation. We thus opted to create new data sets for evaluation.
Table 2 shows sample data from these datasets.

Standard Science Analogies (std): As far as we could find, the closest dataset
consisting of conceptual analogies is from Turney (2008). It consists of ten
standard science analogies. However, these only contain the source and target
concepts but not any explanation in natural language.

Science analogies from academic Q&A sites saqa: We searched for quiz questions
that asked to create analogies on academic Q&A sites like Chegg.com, Study.com
555https://chegg.com/, https://study.com/. We manually inspected the data and
found no personal identifiers or offensive content. We manually compiled the
datasets, no scraping was done. by using search queries like ‘create an
analogy’, ‘analogy to explain’, and manually downloaded the relevant questions
and answers. After manually removing irrelevant data, 75 unique question-
answer pairs were obtained. Next, we manually extracted the analogies from
answers, i.e., target and source concepts, and the explanation of the
analogical similarity.

There are total 109 concepts (about high-school science) with 148 English
analogies. The average word length of analogies is 62.25 words.

Table 2: Sample analogies from std and saqa. Dataset |  Target |  Source |  Explanation
---|---|---|---
std |  atom |  solar system |  -
saqa |  ligase |  sewing machine |  … Ligase is similar to a sewing machine, as it binds two elements … (25 words)

##  5 Experiment Results

In this section, we present our experiment results and examine each of the
three research questions introduced earlier.

###  5.1 Feasibility Analysis

We first examine RQ1 and investigate whether InstructGPT is capable of
generating analogies with simple prompts by looking at the results on the
smaller std dataset which contains well-known analogies. Here, we seek
standard analogies, so we designed prompts with keywords such as "well-known
analogy", "often used to explain", etc. The full list of prompts is in Table
17, Appendix C).

We observed that all the prompts were successful in retrieving natural
language analogies to some extent but they differed in several aspects. Table
LABEL:tab_sample_std shows sample analogies generated by two of our prompts
(P7 and P2, Table 17) for the target concept “natural selection.” In this
case, the reference answer in the std dataset is “artifical selection,” which
P7 successfully retrieved, while P2 generated a different but also valid
analogy. Such variations indicate both the potential of using different
prompts to generate (multiple) different analogies and the model sensitivity
to prompt design, which we further investigate in Section 5.2.

To quantify the effectiveness of different prompts, we manually evaluated the
source concepts mentioned in the generated analogies (if any). Table 3 shows
the number of exact matches of generated source concepts to those in the
reference std dataset, along with the number of “valid” source concepts
generated. Valid means a reasonable analogy that is either commonly known
(e.g., easily available on the internet 666Note that commonly known does not
necessarily mean available on the internet. We use it only as a proxy here
since there is no good way to determine what is common knowledge.) or contains
a meaningful justification. All prompts generated valid analogies in most
cases, even if they didn’t exactly match the reference source concept further
suggesting the promise of InstructGPT for generating meaningful analogies.
Note that the low number of exact matches with the reference dataset is
expected to some extent because there are several possible “valid” analogies
for a given source concept and so there is a small chance that the model would
generate exactly the same analogous concept as in the reference.

Table 3: Number of analogies that match the ground truth or are otherwise meaningful, out of the total ten analogies generated for std target concepts by the seven prompts (P1-P7). | P1 | P2 | P3 | P4 | P5 | P6 | P7
---|---|---|---|---|---|---|---
# Match | $3$ | $3$ | $6$ | $4$ | $3$ | $5$ | $3$
# Valid | $6$ | $9$ | $9$ | $8$ | $7$ | $10$ | $10$

###  5.2 Robustness analyses

As observed in many other applications of prompted PLMs, the performance of a
task tends to be sensitive to the prompts used and the temperature parameter
Lu et al. (2021); Zhao et al. (2021). Moreover, many PLMs are known to be
vulnerable to the presence of spelling errors Pruthi et al. (2019); Ma et al.
(2020). Thus, it is important to experiment with variations of both the
prompts and the temperature parameter (with frequency_penalty, Section 5.2.2
), and spelling errors and study how they impact the generated analogy (RQ2).

For these analyses, we need to compare the model performance in a large number
of configurations, which makes human evaluation impossible. Thus, we rely on
automatic metrics. Automatic evaluation of natural language generation is
known to be challenging (e.g., long-form question answering Krishna et al.
(2021)) and automatic metrics generally have low correlation with human
judgment Callison-Burch et al. (2006); Raffel et al. (2019). Evaluation of
analogies is even more challenging especially because a target concept could
have several valid analogies with seemingly different meanings (e.g.,
“artificial selection” vs. “sieve” from Section 5.1). Thus, before using
existing methods, we designed sanity checks and found that those methods
behave as we expect (e.g., analogies have a higher score than non-analogies,
see Appendix B). We note that our sanity checks are only the necessary and not
the sufficient requirements of a good metric for evaluating analogies as they
do not evaluate creativity or reasoning. However, we use them as an
approximation only for relative comparison between methods on the same task as
they are unlikely to favor any single method.

We use three representative measures of automatic evaluation of generated
text: bleurt Sellam et al. (2020) (b), meteor Lavie and Agarwal (2007) (m),
rouge-l (r)777https://pypi.org/project/rouge-score/ Lin
(2004)888https://www.nltk.org/api/nltk.translate.meteor_score.html. bleurt (b)
is used as the primary metric for evaluation since it is a recent machine
learning-based metric that has been shown to capture semantic similarities
between texts Sellam et al. (2020).

Similar average bleurt values would indicate that the prompts are equally good
(or bad) on a task, but not necessarily in the same way. On the other hand,
Kendall’s Tau Kendall (1938) indicates how well the ranks of two variables are
correlated. This would suggest that those prompts have similar strengths and
weaknesses. Thus, we analyze both scores to get a more complete picture of
hyperparameter sensitivity.

####  5.2.1 Analysis of prompts

To study the effectiveness and robustness of different prompts for analogy
generation in the unsupervised setting, we manually designed several prompts
for all the problem settings. The different prompt variants are all
paraphrases that are semantically similar. The main ways they differ are: 1.
Questions vs. Imperative Statements (e.g., P5 vs. P2, Table 5); 2. Synonyms
(e.g., P2 vs. P3, Table 5); 3. Word Ordering (e.g., P1 vs. P3, Table 4). We
only study the zero-shot setting mainly because the choice/number of examples
in few-shot could make an impact on the generated analogies and make it harder
to interpret our experiment results.

Table 4: Prompts for no_src Id |  Prompt
---|---
P1 |  Explain <target> using an analogy.
P2 |  Create an analogy to explain <target>.
P3 |  Using an analogy, explain <target>.
P4 |  What analogy is used to explain <target>?
P5 |  Use an analogy to explain <target>.

Prompts for the no_src and wsrc settings are in Tables 4,5, respectively.
Here, <target>, <src> are target and source concept placeholders.

Table 5: Prompts for wsrc Id |  Prompt
---|---
P1 |  Explain <target> using an analogy involving <src>.
P2 |  Explain how <target> is analogous to <src>.
P3 |  Explain how <target> is like <src>.
P4 |  Explain how <target> is similar to <src>.
P5 |  How is <target> analogous to <src>?
P6 |  How is <target> like <src>?
P7 |  How is <target> similar to <src>?

Our major findings are as follows:

Questions and statements are significantly different: The question prompts are
P4, Table 4 and P5-P7, Table 5. From Tables 6 and 7, questions have
significantly different and lower scores than statements. This could be an
artifact of how the InstructGPT models were trained and should be further
investigated.

Table 6: Comparison of performances of different prompts and temperatures in no_src. $*$ and $\dagger$ mean statistically significant compared to the best performing setting at p<0.1 and p<0.05 respectively based on a two-tailed t-test. | B | R | M
---|---|---|---
P1tl | $0.46$ | $0.187$ | $0.154$
P1th | $0.448^{\dagger}$ | $0.181^{\dagger}$ | $0.167$
P2tl | $0.451$ | $0.193$ | $0.154$
P2th | $0.45^{*}$ | $0.184$ | $0.161$
P3tl | 0.462 | 0.196 | $0.164$
P3th | $0.452$ | $0.188$ | 0.171
P4tl | $0.427^{\dagger}$ | $0.170^{\dagger}$ | $0.126^{\dagger}$
P4th | $0.431^{\dagger}$ | $0.179^{\dagger}$ | $0.156$
P5tl | $0.451$ | $0.188$ | $0.154$
P5th | $0.449^{*}$ | $0.183^{*}$ | $0.163$

Impact of synonyms and word order: Prompt performances vary based on synonyms
and word order. For example, some synonymous prompt pairs (e.g, P2-P4, P5-P7
in wsrc) are more correlated than others (e.g., P2-P3, P5-P6 in wsrc). This
could be because “analogous to” and “similar to” share a word unlike the other
synonym “like”. As expected, prompts with the most different meanings (e.g.,
P1 in wsrc – involving <src> is not necessarily the same as analogous to
<src>) are least correlated with others. However, from Table 7, the average
performances of synonymous prompts (e.g., $P2_{tl}$ and $P3_{tl}$, $P2_{tl}$
and $P5_{tl}$) are not significantly different. Overall, this suggests that
InstructGPT is more robust to synonyms/word-order than to the prompt style
(question/imperative statements) for this task. The overall best-performing
prompts (P3 in no_src, P2 in wsrc) contain some form of the word “analogy”
rather than its synonyms, confirming that precise and direct prompts are
better.

Table 7: Comparison of performances of different prompts and temperatures in wsrc. $*$ and $\dagger$ mean statistically significant at p<0.1 and p<0.05 compared to the best performing setting respectively based on a two-tailed t-test. | B | R | M
---|---|---|---
P1tl | $0.504$ | $0.223$ | $0.187^{\dagger}$
P1th | $0.497^{\dagger}$ | $0.212^{\dagger}$ | $0.199$
P2tl | 0.515 | $0.217$ | $0.203$
P2th | $0.502^{*}$ | $0.210^{\dagger}$ | 0.208
P3tl | $0.504$ | 0.229 | $0.191$
P3th | $0.504$ | $0.216$ | $0.203$
P4tl | $0.506$ | $0.214$ | $0.197$
P4th | $0.497^{\dagger}$ | $0.206^{\dagger}$ | $0.2$
P5tl | $0.499^{*}$ | $0.217$ | $0.18^{\dagger}$
P5th | $0.496^{\dagger}$ | $0.211^{\dagger}$ | $0.191^{*}$
P6tl | $0.500^{*}$ | $0.216$ | $0.176^{\dagger}$
P6th | $0.494^{\dagger}$ | $0.212^{\dagger}$ | $0.183^{\dagger}$
P7tl | $0.497^{\dagger}$ | $0.208^{\dagger}$ | $0.179^{\dagger}$
P7th | $0.492^{\dagger}$ | $0.204^{\dagger}$ | $0.186^{\dagger}$
Figure 1: Kendall’s Tau correlation between bleurt scores of various prompts
and temperatures in wsrc

####  5.2.2 Analysis of temperature

Higher temperature increases the randomness in the generated text and is often
suggested for creative tasks Lucy and Bamman (2021). Since some analogies
require creativity, we are especially interested in studying the impact of
this hyperparameter.

We explore two settings. Low Temperature (tl): this is a deterministic
setting, where temperature = frequency_penalty = presence_penalty = 0. High
Temperature (th): Here temperature is set to 0.85. To avoid repetition of
words and topics, we set frequency_penalty = 1.24 and presence_penalty = 1.71.
These hyperparameters were selected based on initial qualitative exploration.
To account for the randomness, we set best_n = 3, i.e., select the best
response out of three generated responses, and generate 5 such best responses.
In all experiments, we report the average performance of all 5 responses.

From the results in lower-right corner of Figures 1 and 3 (in Appendix D), we
make the following two observations:

Firstly, at high temperature, prompts are generally well-correlated with each
other suggesting lesser sensitivity to prompt design at high temperatures.
This requires further investigation because we expect higher randomness to
generate a variety of different analogies, and thus have lower correlations in
general.

Secondly, the overall performances of the high-temperature variants are
generally lower than their low-temperature counterparts. To investigate when
high temperature could help, we further looked into a case in the wsrc setting
where the high-temperature version of the best prompt, ($P2_{th}$), performed
much better. The results are shown in Table 8. In this case, unlike $P2_{hl}$,
$P2_{tl}$ fails on identifying the target and also generates incorrect facts,
(“rubber of your lungs”). This shows some evidence of high temperature prompts
working better for more complex and creative analogies, which should be
investigated further in the future.

Table 8: Analogies generated in the high and low-temperature setting by the prompt: Explain how breathing mechanism of frogs is analogous to blowing of the balloon. Temp. |  Generated Analogy
---|---
High |  … When you blow into a balloon, the air pressure … The frog’s respiratory system works in a similar way. When you breathe in, you increase the air pressure inside your lungs. This increased air pressure causes the walls …
Low |  When you blow up a balloon, the air inside the balloon is pushing outwards on the rubber of the balloon. This is the same thing that happens when you breathe in: the air inside your lungs is pushing outwards on the rubber of your lungs.

####  5.2.3 Analysis of synthetic spelling errors

Since obtaining natural spelling errors can be difficult, injecting synthetic
errors is a common way to test the robustness of models (e.g., Jayanthi et al.
(2020). Thus, following previous work Sakaguchi et al. (2017); Jayanthi et al.
(2020), we injected the following four types of character-level errors to the
internal characters of the target concept in the prompt: Delete (delete one
randomly chosen character), Permute (switch two randomly chosen adjacent
characters in the string), Insert (insert one random alphabet at a random
position), and Replace (replace one randomly chosen character in the string
with a random alphabet). Target concepts with length less than 3 were kept
unchanged.

Average bleurt scores from three different runs for all prompts in the low-
temperature setting in no_src are shown in Table 9. Overall, the performance
decreases, indicating the sensitivity of language models to spelling errors.
Further, Replace generally leads to the biggest performance drop for all
prompts ($\sim 3-7\%$ relative decrease). The model is generally most robust
to Insert, similar to the results reported in previous work on word
recognition using neural networks Sakaguchi et al. (2017).

Table 9: Impact of injecting Delete (D), Permute (P), Insert (I) and Replace (R) errors to the target concept in the prompt compared to the original (O) prompt based on bleurt scores. $*$ and $\dagger$ mean statistically significant at p<0.1 and p<0.05 respectively based on a two-tailed t-test. | D | P | I | R | O
---|---|---|---|---|---
P1 | $0.438^{\dagger}$ | $0.437^{\dagger}$ | $0.436^{\dagger}$ | $0.429^{\dagger}$ | $0.46$
P2 | $0.431^{\dagger}$ | $0.434^{\dagger}$ | $0.442$ | $0.427^{\dagger}$ | $0.451$
P3 | $0.444^{\dagger}$ | $0.445^{\dagger}$ | $0.447^{*}$ | $0.44^{\dagger}$ | $0.462$
P4 | $0.423$ | $0.424$ | $0.428$ | $0.416$ | $0.427$
P5 | $0.438^{*}$ | $0.437^{*}$ | $0.441$ | $0.435^{\dagger}$ | $0.451$

###  5.3 Analysis of model size

Finally, we examine RQ3, i.e., how does the model size impact the quality of
the generated analogies. In general, models with more parameters can be
expected to perform better. We now study whether the same holds for this task
and how much the model size impacts the performance.

Figure 2 shows the bleurt scores of various models on both the task setups. As
expected, the performance increases significantly with model size in both wsrc
and no_src, suggesting that larger models are better at generating analogy-
like text for the given targets. Further, the biggest improvement is seen as
the number of parameters increases from 0.3B to 1.3B in both settings (19.17%
and 15.34% relative improvements, respectively).

Similar to what we observed in the case of the 175B Davinci model, the
performance in wsrc is higher than that in no_src for other models too. This
confirms that all models have some capacity to incorporate the source provided
in the prompt.

Figure 2: Average performances of various InstructGPT models based on bleurt
scores.

###  5.4 Human evaluation

To further validate the generated analogies more comprehensively, we also
conducted human evaluation as described below.

####  5.4.1 Annotation Setup

We conducted the study on Amazon Mechanical Turk. Based on manual evaluation
of responses to screening tests (Appendix E), we selected 17 workers for the
main study.

Further, we created a sample dataset for evaluating analogies generated both
in the no_src and wsrc settings. In total, we generated 13k analogies 9996
analogies (5 in high temperature and 1 low temperature) *109 target concepts*5
prompts*4 models in no_src and 18k analogies 1010106 analogies (5 in high
temperature and 1 low temperature) *109 target concepts*7 prompts*4 models in
wsrc. From this data, we randomly selected 42 concepts for the no_src setup
and 21 of them were selected for the wsrc setup (to have comparabale number of
analogies in both settings). The analogies for the selected concepts,
generated by all the models using all the prompts in the low temperature
setting were selected for evaluation since low temperature was better based on
automatic evaluation.

In total, 1407 unique analogies (576 from wsrc, 770 from no_src, and 61 human-
generated from saqa) were evaluated by 3 workers each, which is common in
previous work on evaluation of automatically generated text van der Lee et al.
(2021). The main study had one question asking workers to evaluate whether the
shown candidate analogy was meaningful for the target concept (Yes/No/Can’t
decide) and provide a text input for explaining their choice (Figure 6,
Appendix F). Please refer to Appendix E for more details of the study design.

####  5.4.2 Quantitative Results

Table 10 shows the percentage of analogies rated as meaningful, based on
majority vote, for the various models and the human references from saqa.
There were <2% ties or cases with ‘Can’t decide’ as the majority, which were
discarded. The Fleiss’ kappa Fleiss (1971) inter-annotator agreement was 0.347
in case of wsrc (plus human references for the selected concepts for wsrc
concepts), indicating fair agreement and 0.553 in case of no_src (plus human
references for the selected concepts for no_src concepts) indicating moderate
agreement.

We observe that the percentage of meaningful analogies increases with model
size, again confirming that larger models have a higher capacity to generate
analogies. Interestingly, in the no_src setting, the largest model has
comparable performance to humans. We note that this doesn’t necessarily mean
that those models are creative or have commonsense reasoning skills as they
could have simply memorized those analogies, which a known problem of such
models Bender et al. (2021). It requires further research to test whether the
models generate novel analogies unseen during training.

Moreover, upon inspection, we found that the human-generated analogies
sometimes had minor issues, such as grammatical errors, which could impact
their rating by annotators. So, it is possible that analogies written by
experts, such as science instructors proficient in English, might be rated
higher. Nevertheless, these results are quite encouraging as the model seems
to have comparable performance to general online users who wrote the analogies
in our reference dataset.

In the wsrc setting, the performance of InstructGPT is lower than human
performance. This could be because there is a lesser likelihood of seeing the
exact same analogy, i.e., the one asked to explain in the prompt, during
training, compared to seeing any analogies for the target concept as required
in the no_src setting. So, wsrc might require more “analogical reasoning” from
the models, especially for explaining analogies not seen during training. This
highlights the importance of human evaluations for such tasks because
otherwise, based on automatic evaluation alone, we would conclude that this is
an easier setting. This is because metrics like bleurt cannot assess the
soundness of the generated reasoning.

We also compute the no_src performance on the 21 shared concepts (no_src21,
Table 10) for a fair comparison between the two settings. It is interesting to
note that the performances of smaller models increase while that of larger
models go down in the wsrc setting. This could be because the provided source
in the prompt helps provide some guidance to the smaller models. For example,
even by copying parts of the prompt (i.e., source and target), they could
generate meaningful analogies (e.g., <source> is like <target>) in a few
cases. Since their performance in the no_src setting is very poor, even minor
help or “tricks” would lead to performance improvement. On the other hand, the
larger models that already performed very well, likely do not have much to
gain from such help and, in fact, perform worse due to the analogical
reasoning argument made above.

Overall, this highlights some limitations of the InstructGPT model for
analogical reasoning, which requires further research for improvement.

####  5.4.3 Error Analysis

The annotators were also asked to explain their answer choice (i.e, meaningful
analogy or not). By inspection, we identified the following major themes based
on the workers’ explanations for choosing “not meaningful” across all
models/tasks. These themes are not mutually exclusive and multiple themes were
often found for one wrong generation.

1\. No Analogy: This is one of the most common cases where the model failed to
generate any analogy at all. Instead, it mostly generated a simple
description/definition of the target concept. In a few cases, it also
generated a tautology or an example. For example, “The b-lymphocytes are
similar to the white blood cells.”

2\. Irrelevant to target: The generated text contained little to none relevant
information pertaining to the target. One interesting reason behind this was
capitalization for abbreviations. For example, since the targets in the prompt
were lowercased (e.g., nadh), smaller models were unable to identify
abbreviations, while the larger models succeeded at this. Another reason
observed was that of an ambiguous target, e.g., computer “mouse” misidentified
as a rodent. In more insidious cases, the text looked correct but presented
incorrect facts.

3\. Incorrect source or explanation: Here, important details about the source
concept were either incorrect or missing, or the provided explanation was
insufficient, making the analogy completely wrong or weak at best. For
example, “A molecule of DNA is like a drop of water. It has a specific shape
and size, and it can carry the genetic instructions for making a particular
organism.”

Some error types found in other natural language generations from GPT-3 Dou et
al. (2021), e.g., incoherence and grammar, were also found in our task.
Further research is required to quantify them for analogical generation and
attempt to fix them.

Table 10: Percentage of meaningful analogies generated by various InstructGPT models and humans based on human evaluation. Highest value per row is underlined. | 0.3B | 1.3B | 6.7B | 175B | Human
---|---|---|---|---|---
no_src | 1.90 | 15.61 | 48.29 | 70.05 | 66.67
wsrc | 8.97 | 29.05 | 38.46 | 53.79 | 71.88
no_src21 | 0 | 12.0 | 47.0 | 66.99 | 71.88

##  6 Limitations

A major limitation of our study is that we only studied analogies on a small
reference dataset in one domain (high-school science). Our newly created
reference data sets are relatively small due to limited resources found
online. But, the sample size of the automatically generated analogies we
evaluated was large ($\sim$ 31k automatically evaluated, and $\sim$ 1.4k
manually evaluated) thereby mitigating some concerns about bias due to small
dataset size. Moreover, as our research questions study an open-ended
generation task, having a pre-defined list of reference candidates is not
ideal for evaluation. Thus, future research is required to more thoroughly
evaluate the generated analogies and investigate the generalizability of the
findings to other domains.

Further, the manual evaluation was conducted by a selected group of people in
the US and might not reflect the opinions of a more diverse group. Moreover,
our kappa scores of 0.3-0.5, although common in previous NLG evaluation work
van der Lee et al. (2021), are not on the higher end. In general, thresholds
to determine what counts as high or low kappa scores tend to be open to
interpretation van der Lee et al. (2021). Thus, we’ve released our annotated
and full datasets online, as also suggested in van der Lee et al. (2021), and
invite other researchers to further investigate them.

##  7 Conclusion

In this study, we proposed and studied the novel task of generating analogies
by prompting InstructGPT. Our experiments showed that the InstructGPT is
effective on this task when precise prompts are used, thus offering a
promising new way to generate analogies, which can break the limitation of the
traditional analogy generation methods in requiring a pre-generated structured
representation.

By evaluating the performances of the various designed prompts in multiple
temperature settings and in the presence of synthetic spelling errors, we
found that the InstructGPT model is sensitive to those variations (e.g.,
question vs. imperative-style prompts). Additionally, based on human
evaluation, we found that the quality of the generated analogies substantially
depends on the model size. The largest model was found to achieve human-level
performance at generating analogies for given target concepts. There is still
much room for improvement at the challenging task of explaining the analogical
similarity between the given target and source concepts.

Our work opens up many exciting opportunities for future work both for
application-oriented and foundational research on PLMs for analogy generation.
For example, conducting more robustness analyses based on prompt perturbations
(e.g., natural spelling mistakes, grammar, length, etc.). Also, in addition to
the unsupervised approaches we explored in this paper, it is interesting to
develop supervised approaches for this task including by fine-tuning PLMs on
our created datasets.

##  8 Acknowledgments

This work is supported in part by the IBM-Illinois Center for Cognitive
Computing Systems Research (C3SR) as an IBM AI Horizon’s Network.

##  9 Ethical Considerations

The risks associated with using PLMs for analogy generation are similar to
those of NLG tasks, such as bias, toxicity, and misinformation Bender et al.
(2021); Weidinger et al. (2021). Accordingly, these should be carefully
evaluated before deploying the models for any practical applications, such as
education.

Furthermore, there is a steep monetary and environmental cost associated with
using the GPT-3 models, especially Davinci. The OpenAI API charges $0.06 /1K
tokens. Including early experiments, analogy generation in this study costed a
total of about $240. Since we conducted multiple runs with the same prompt
account for randomness (e.g., in the high temperature setting), the costs rose
sharply.

## References

  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 610–623.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006.  Re-evaluating the role of bleu in machine translation research.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dou et al. (2021) Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021\.  Scarecrow: A framework for scrutinizing machine text.  _arXiv preprint arXiv:2107.01294_.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Forbus et al. (2017) Kenneth D Forbus, Ronald W Ferguson, Andrew Lovett, and Dedre Gentner. 2017.  Extending sme to handle large-scale cognitive modeling.  _Cognitive Science_ , 41(5):1152–1201.
  * Jayanthi et al. (2020) Sai Muralidhar Jayanthi, Danish Pruthi, and Graham Neubig. 2020.  Neuspell: A neural spelling correction toolkit.  _arXiv preprint arXiv:2010.11085_.
  * Kendall (1938) Maurice G Kendall. 1938.  A new measure of rank correlation.  _Biometrika_ , 30(1/2):81–93.
  * Kittur et al. (2019) Aniket Kittur, Lixiu Yu, Tom Hope, Joel Chan, Hila Lifshitz-Assaf, Karni Gilon, Felicia Ng, Robert E Kraut, and Dafna Shahaf. 2019.  Scaling up analogical innovation with crowds and ai.  _Proceedings of the National Academy of Sciences_ , 116(6):1870–1877.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  _arXiv preprint arXiv:2103.06332_.
  * Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007.  Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments.  In _Proceedings of the second workshop on statistical machine translation_ , pages 228–231.
  * Li et al. (2021) Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021.  Pretrained language models for text generation: A survey.  _arXiv preprint arXiv:2105.10311_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _arXiv preprint arXiv:2104.08786_.
  * Lucy and Bamman (2021) Li Lucy and David Bamman. 2021.  Gender and representation bias in gpt-3 generated stories.  In _Proceedings of the Third Workshop on Narrative Understanding_ , pages 48–55.
  * Ma et al. (2020) Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020\.  Charbert: character-aware pre-trained language model.  _arXiv preprint arXiv:2011.01513_.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mitchell (2021) Melanie Mitchell. 2021.  Abstraction and analogy-making in artificial intelligence.  _arXiv preprint arXiv:2102.10717_.
  * Newby et al. (1995) Timothy J Newby, Peggy A Ertmer, and Donald A Stepich. 1995.  Instructional analogies and the learning of concepts.  _Educational Technology Research and Development_ , 43(1):5–18.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019.  Combating adversarial misspellings with robust word recognition.  _arXiv preprint arXiv:1905.11268_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rossiello et al. (2019) Gaetano Rossiello, Alfio Gliozzo, Robert Farrell, Nicolas R Fauceglia, and Michael Glass. 2019.  Learning relational representations by analogy using hierarchical siamese networks.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3235–3245.
  * Sakaguchi et al. (2017) Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme. 2017.  Robsut wrod reocginiton via semi-character recurrent neural network.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  In _Proceedings of ACL_.
  * Turney (2008) Peter D Turney. 2008.  The latent relation mapping engine: Algorithm and experiments.  _Journal of Artificial Intelligence Research_ , 33:615–655.
  * Ushio et al. (2021) Asahi Ushio, Luis Espinosa-Anke, Steven Schockaert, and Jose Camacho-Collados. 2021\.  Bert is to nlp what alexnet is to cv: Can pre-trained language models identify analogies?  _arXiv preprint arXiv:2105.04949_.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021\.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_.
  * Zhao et al. (2021) Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  _arXiv preprint arXiv:2102.09690_.

##  Appendix A Hyperparameters

Based on initial explorations, where we varied the number of maximum tokens
between 0 and 1000 in increments of 100, and then from 935-955 in increments
of 1, we noticed that setting a high number of maximum tokens worked better in
generating more comprehensive analogies that were not abruptly cut-off and
there was little sensitivity to higher values around 950. So, we randomly
chose one value in that range (939). The default value of top_p = 1 was used.

##  Appendix B Suitability of existing evaluation metrics

To first investigate the suitability of existing evaluation metrics for
generated analogies before we can trust any evaluation results using them, we
designed two testers to examine whether the existing metrics behave as
expected: 1) Ordering Tester ot: This tester is to see if an evaluation metric
can order a set of methods that have known orders between them correctly as
expected. 2) Random Perturbation Tester rpt : This tester checks if an
evaluation metric responds to a random perturbation to the ground truth data
used for evaluation. A reasonable metric is expected to generate lower
performance figures after perturbation.

We use those two testers to study the suitability of three popular and
representative measures of automatic evaluation of generated text: bleurt
Sellam et al. (2020), meteor Lavie and Agarwal (2007), rouge-l Lin (2004).

bleurt (b) is a recent machine learning-based metric that has been shown to
capture semantic similarities between text. rouge-l
(r)111111https://pypi.org/project/rouge-score/ measures longest matching
subsequence of words. We use its F1-score. meteor (m)
121212https://www.nltk.org/api/nltk.translate.meteor_score.html matches word
stems and synonyms also.

Design of testers: We design an ot and a rpt based on the following baseline
methods:

No Analogy baseline (no_anlgy): Here, the prompts instruct the model to
generate an explanation or description of the target concept and do not ask
for an analogy explicitly. Thus, we expect the generated text to be in a
different “style” than analogies and the overall performance to be lower.
However, the generation would still contain other relevant keywords describing
the target. Thus, it is a good baseline to test if the metrics can distinguish
between analogies and other descriptions.

Random baselines: For each of the three setups, we introduced random baselines
(no_anlgy_rand, no_src_rand, and wsrc_rand, respectively) where a generated
string is evaluated against a random analogy (excluding the correct matching
analogy) in the reference dataset (i.e., applying a random perturbation to the
ground truth). These baselines preserve the “style” of the text but not the
content. We expect these methods to perform worse than their non-random
counterparts.

Additionally, no_src setting is expected to perform worse than wsrc because in
wsrc, the model has more information (i.e., the source concept) and thus has
better chances of generating the correct analogical explanation. Thus, the
expected order is no_anlgy < no_src < wsrc.

Metric testing results: Table 11 shows the overall results of experiments on
the saqa dataset using the Davinci model. Each row shows the highest average
scores given by a metric in various setups (performances of each prompt are in
Section 5.2 and at the end of this section.).

We can see that all the three metrics order the setups as expected, i.e.,
random baselines are assigned a lower score than non-random setups, and scores
for no_anlgy < no_src < wsrc. This suggests that all the three metrics have
“passed" our two testers and thus can be reasonably used to evaluate whether
the automatically generated analogies are similar to those generated by
humans. In other words, they should help assess whether the generated text is
relevant to the target concept and discuss properties of the concept that
could be explained using analogies (because they passed rpt), and written in
an analogical style (because they passed ot).

Moreover, the results also indicates that the InstructGPT model is able to
follow the prompts in the three settings to some extent and generate non-
analogical descriptions, general analogies, and analogies containing the
source concepts, in those settings respectively.

In terms of discernment power, all metrics have small gaps between the scores
of random and non-random settings. Similar results were previously reported in
Krishna et al. (2021) for rouge scores on long-form question-answering. Out of
the three metrics, the bleurt score has the largest gaps in all the settings,
both between the random and non-random baselines and also between settings. It
is also shown to capture semantic similarity well Sellam et al. (2020). Thus,
we use it as the main metric in the rest of the experiments.

Table 11: Testing results using ot and rpt. The higher score between the random baseline and the non-random setup is bolded. Highest score in a row in underlined. | no_anlgy_rand | no_anlgy | no_src_rand | no_src | wsrc_rand | wsrc
---|---|---|---|---|---|---
B | 0.349 | 0.445 | 0.375 | 0.462 | 0.385 | 0.515
R | 0.122 | 0.183 | 0.132 | 0.196 | 0.122 | 0.229
M | 0.099 | 0.158 | 0.109 | 0.171 | 0.109 | 0.208
Table 12: Prompts for no_anlgy Id |  Prompt
---|---
P1 |  Explain <target>.
P2 |  What is <target>?
P3 |  Explain <target> in plain language to a second grader.
Table 13: Comparison of performances of different prompts and temperatures in no_anlgy. | B | R | M
---|---|---|---
P1tl | 0.434 | 0.183 | 0.149
P1th | 0.432 | 0.18 | 0.158
P2tl | 0.43 | 0.175 | 0.129
P2th | 0.425 | 0.172 | 0.136
P3tl | 0.445 | 0.180 | 0.132
P3th | 0.444 | 0.179 | 0.144
Table 14: Comparison of performances of different prompts and temperatures in no_src_rand. | B | R | M
---|---|---|---
P1tl | 0.375 | 0.132 | 0.103
P1th | 0.367 | 0.123 | 0.108
P2tl | 0.359 | 0.116 | 0.092
P2th | 0.366 | 0.127 | 0.105
P3tl | 0.362 | 0.124 | 0.099
P3th | 0.364 | 0.126 | 0.109
P4tl | 0.338 | 0.115 | 0.084
P4th | 0.348 | 0.121 | 0.1
P5tl | 0.358 | 0.121 | 0.097
P5th | 0.348 | 0.122 | 0.107
Table 15: Comparison of performances of different prompts and temperatures in wsrc_rand. | B | R | M
---|---|---|---
P1tl | 0.37 | 0.120 | 0.094
P1th | 0.363 | 0.122 | 0.107
P2tl | 0.385 | 0.117 | 0.096
P2th | 0.381 | 0.12 | 0.109
P3tl | 0.358 | 0.117 | 0.095
P3th | 0.359 | 0.115 | 0.1
P4tl | 0.367 | 0.113 | 0.096
P4th | 0.37 | 0.115 | 0.105
P5tl | 0.36 | 0.113 | 0.09
P5th | 0.356 | 0.117 | 0.094
P6tl | 0.346 | 0.111 | 0.086
P6th | 0.347 | 0.113 | 0.091
P7tl | 0.353 | 0.114 | 0.092
P7th | 0.352 | 0.109 | 0.093
Table 16: Comparison of performances of different prompts and temperatures in no_anlgy_rand. | B | R | M
---|---|---|---
P1tl | 0.346 | 0.115 | 0.087
P1th | 0.349 | 0.122 | 0.099
P2tl | 0.322 | 0.116 | 0.077
P2th | 0.327 | 0.113 | 0.081
P3tl | 0.334 | 0.111 | 0.079
P3th | 0.336 | 0.11 | 0.081

##  Appendix C Experiments on std dataset

Table 17: Prompts for std analogies Id |  Prompt
---|---
P1 |  Explain <target> using an analogy.
P2 |  Explain <target> using a well-known analogy.
P3 |  What analogy is often used to explain <target>?
P4 |  Using a well-known analogy, explain <target>.
P5 |  Using an analogy, explain <target>.
P6 |  What is a well-known analogy to explain <target>?
P7 |  What is analogous to <target>?
Table 18: Most common analogies generated for each target concept in the std dataset. #Pmt. means number of prompts that generated the shown analogy. Target | Most common src. | # Pmt.
---|---|---
mind | computer | 7
atom | solar system | 6
heat transfer | fluid/water flow | 4
sounds | wave | 4
respiration | combustion | 3
light | river | 3
planet | rock | 2
bacterial mutation | game of telephone | 3
natural selection | sieve | 2
gas molecules | balls | 2

##  Appendix D Experiments on saqa dataset

Figure 3: Kendall’s Tau correlation between BLEURT scores of various prompts and temperatures in no_src Table 19: Comparison of lengths of generated responses by question (Q) vs. statement (S) in the wsrc setting. Question versions of the prompts generate fewer words on average, than their statement counterparts. Prompt Pair | Avg. Len. (S) | Avg. Len. (Q)
---|---|---
P2-P5 | 43.93 | 34.53
P3-P6 | 32.55 | 31.4
P4-P7 | 42.51 | 32.72
Table 20: Comparison of lengths of generated responses by low and high temperatures in the no_src setting. High temperature generates consistently longer analogies. Same trend is observed in other settings also. Prompt | Avg. Length (tl) | Avg. Length (th)
---|---|---
P1 | 39.74 | 47.62
P2 | 32.67 | 40.71
P3 | 40.06 | 46.62
P4 | 32.51 | 40.13
P5 | 36.53 | 38.50

##  Appendix E Mturk study details

For identifying qualified workers on Amazon Mechanical Turk, we designed a
pre-screening test (Mturk Qualification) asking them to identify the
meaningful analogy for a target concept (Figure 4, Appendix F). Further, we
used the following additional qualifications: workers should have completed at
least 5k tasks with >98% approval rate and be located in the US since the task
requires proficiency in english (this way of filtering is not perfect but
there is currently no good way to identify native english speakers via Mturk).
We did not collect any other demographic or geographic information about the
workers.

Those who passed these qualifications worked on a small test batch of
analogies asking detailed questions about their quality (Figure 5, Appendix
F). The questions consisted of both Likert-style or Binary choice questions
and text inputs asking them to explain their choices. We manually assessed
their responses, especially paying close attention to their reasoning to
identify qualified workers for the main study.

For both the main study and the screening, a simple definition of the target
from sites like Simple English Wikipedia
131313https://simple.wikipedia.org/wiki/Main_Page was provided to workers as
reference and they were encouraged to refer to the internet to learn more
about the shown concepts. We also provided several sample annotations as part
of the instructions to guide workers. Moreover, we were available to answer
clarification questions via a shared chatroom.

Annotators were paid at the rate of $50/hr. The rate was decided based on open
discussions with them and is above the minimum wage. They were informed that
the data generated would be used for research purposes. We consulted with our
university ethics board and found that IRB was not required for this study.

##  Appendix F Human evaluation interface

Figure 4: Pre-screening question for identifying qualified workers. Figure 5:
Sample interface for screening qualified workers. Figure 6: Sample interface
for human evaluation of the analogies.


#  NSina: A News Corpus for Sinhala

###### Abstract

The introduction of large language models (LLMs) has advanced natural language
processing (NLP), but their effectiveness is largely dependent on pre-training
resources. This is especially evident in low-resource languages, such as
Sinhala, which face two primary challenges: the lack of substantial training
data and limited benchmarking datasets. In response, this study introduces
NSina, a comprehensive news corpus of over 500,000 articles from popular
Sinhala news websites, along with three NLP tasks: news media identification,
news category prediction, and news headline generation. The release of NSina
aims to provide a solution to challenges in adapting LLMs to Sinhala, offering
valuable resources and benchmarks for improving NLP in the Sinhala language.
NSina is the largest news corpus for Sinhala, available up to date.

Keywords: news corpus, low-resource languages, text classification, text
generation

NSina: A News Corpus for Sinhala

Hansi Hettiarachchi1, Damith Premasiri2, Lasitha Uyangodage3,
---
Tharindu Ranasinghe4
1Birmingham City University, UK, 2Lancaster University, UK,
3University of Münster, Germany, 4Aston University, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The recent emergence of large language models (LLMs) has ushered in
significant advancements in the field of natural language processing (NLP)
Devlin et al. (2019). These LLMs have produced state-of-the-art results in
many NLP benchmarks, outperforming previous machine learning models such as
LSTMs Lin et al. (2022). Beyond academic research, these recent LLMs, such as
GPT Brown et al. (2020), have also driven the development of widely popular
products, including chatbots, machine translation and writing assistants,
among other applications, making them highly popular among the general public.

While LLMs have achieved considerable success and garnered popularity, their
effectiveness heavily relies on access to resources for weight pre-training.
Consequently, these LLMs excel in high-resource languages but encounter
challenges when applied to low-resource languages Wang et al. (2020). Two
primary factors contribute to the complexities of deploying LLMs in low-
resource linguistic contexts: (1) Limited availability of open-access corpora
for pre-training the models in low-resource languages. and (2) The absence of
appropriate benchmarking datasets in these languages to assess the performance
of the models. In this research, we address these challenges for Sinhala by
releasing NSina: A large News Corpus for Sinhala accompanied by a range of
benchmarking tasks.

Sinhala is an Indo-Aryan language spoken by over 17 million people in Sri
Lanka. Sinhala is one of the two official languages in Sri Lanka.
Predominantly, the Sinhala-speaking community comprises the Sinhalese people,
the largest ethnic group on the island. Despite its sizable community, Sinhala
remains relatively under-resourced compared to other languages spoken in the
region De Silva (2019). The complexities regarding LLMs and low-resource
languages that we discussed previously appear in Sinhala too. While there
exist several multilingual language models, such as XLM-RoBERTa Conneau et al.
(2020), which offer support for Sinhala, the multilingual corpora used to
train these models contain a relatively limited proportion of Sinhala compared
to other languages Wang et al. (2020). For example, the OSCAR 23.01
multilingual corpus Abadji et al. (2022), which has been used to train
multilingual language models, only contains 2.6GB Sinhala text which does not
even contribute to 1% of the total dataset. Furthermore, since these data were
automatically extracted from Common Crawl dumps, they contain noise resulting
from boilerplate content extracted from headers, footers and sidebars of web
crawls Abadji et al. (2022).

There is also a pre-trained BERT model trained specifically for Sinhala
Dhananjaya et al. (2022). However, it’s worth noting that this model has been
trained on a relatively small Sinhala corpus, leading to its inability to
consistently outperform multilingual LLMs in various Sinhala NLP tasks, as
evidenced in both Dhananjaya et al. (2022) and Ranasinghe et al. (2024). These
constraints primarily stem from the limited availability of sizable Sinhala
corpora for model training. Furthermore, as we mentioned before, there is a
huge limitation of available benchmarking datasets for Sinhala. This is also
evident in Dhananjaya et al. (2022), where they evaluated the pre-trained BERT
model in only three text classification tasks.

In this paper, we lay the foundation to address these challenges. Firstly, we
compiled a large news corpus with more than 500,000 news articles from ten
popular news websites in Sri Lanka. Secondly, we compose three NLP tasks from
the news corpus, including two text classification tasks: (1) News media
identification (2) News category prediction and one text generation task: (3)
News headline generation. We release separate train and test sets for each
task, which are sampled from NSina. These datasets can be used as benchmarks
to evaluate the LLMs in Sinhala. While there are several news corpora for
Sinhala Upeksha et al. (2015), NSina is the most updated and the largest
Sinhala news corpus available.

Our main contributions are;

  1. 1.

We introduce NSina, a large News Corpus for Sinhala, which comprises 506,932
news articles, and we describe the steps taken to compile it.

  2. 2.

We introduce three NLP tasks associated with NSina. (i) News media
identification, (ii) News category prediction, and (iii) News headline
generation. We release train and test sets for each task, which are subsampled
from NSina.

  3. 3.

We evaluate several machine learning models on each task and compare the
performance.

  4. 4.

We release NSina, as an open-access publicly available dataset alongside the
trained machine-learning models for each task111The dataset is available at
https://github.com/Sinhala-NLP/NSINA.

## 2\. Dataset Construction

We first present the data collection methodology we used followed by a
detailed statistical analysis of NSina.

### 2.1. Data Collection

First, we identified ten news sources that are popular in Sri Lanka. These
sources encompass a diverse range, including “Adaderana," “ITN News,"
“Lankatruth," “Divaina," “Hiru News," “Lankadeepa," “Vikalpa," “Dinamina," and
“Siyatha." Importantly, we took deliberate steps to ensure a balanced
representation within this selection, encompassing both pro-government and
anti-government news outlets.

We used a Python script to scrape each news media website. For each news
article, we extracted the source, timestamp, headline, news content, URL and
category.

### 2.2. NSina

After data collection, we performed data cleaning. There were a considerable
amount of news articles that had less than ten Sinhala words. These articles
were identified and filtered. The final dataset consisted of 506,932 news
articles. The source news media of these news articles is shown in Table 1. As
can be seen, “Lankadeepa" and “Hiru News" contribute most to the corpus,
having more than 100,000 articles each.

Source | Amount
---|---
Adaderana | 83918
ITN News | 30777
Lankatruth | 48180
Divaina | 26043
Hiru News | 130729
Sinhala News LK | 20371
Lankadeepa | 141663
Vikalpa | 14309
Dinamina | 7642
Siyatha | 3300
Total | 506932

Table 1: Number of news articles from each source in NSina.

Figure 1 shows the token frequency distribution of the news content after
removing the outliers (very long documents). It is clear that most news
articles contain around 60-120 tokens. Furthermore, in Figure 2, we plot the
token frequency of headlines. The figure shows that most of the titles are
short, having between 6-12 tokens. The corpus has more than 1 million tokens
and more than 100,000 unique tokens.

Previous Sinhala news corpus; SinMin Upeksha et al. (2015) was only 1.01 GB in
size. Compared to that, NSINa is 1.87 GB and provides a larger and updated
news corpus. Furthermore, the Sinhala portion of the OSCAR 23.01 corpus Abadji
et al. (2022) only has 301,066 documents and compared to that NSINa has more
than 500,000 properly cleaned Sinhala documents.

Figure 1: Token frequency distribution of news content in NSina Figure 2:
Token frequency distribution of news headline in NSina

## 3\. Tasks

We compiled the following three tasks from NSina. We believe that creating
benchmarks would help to evaluate the LLMs in Sinhala NLP tasks.

  1. 1.

News media identification

  2. 2.

News category prediction

  3. 3.

News headline generation

Each of the tasks had different train/ test sets and machine learning models.
The following sub-sections will describe tasks, machine learning models
trained for the task and the results.

### 3.1. News Media Identification

The first task we compiled from NSina is identifying news source given the
news content. This task serves a dual purpose: firstly, it aids in recognising
the distinctive style of each news outlet in their news presentation.
Moreover, this task can be further explored to unveil potential political
biases within Sri Lankan news media. This is a text classification task, where
the input to the ML models would be the news content, and the expected output
of the model is the news media name.

#### 3.1.1. Data

As all the news instances in NSina contained its news source, constructing the
train/ test set for this task was straightforward. However, as can be seen in
Table 1, since some news media sources have more instances, we undersampled
them. We only used 10,000 instances from each news source. For the two sources
that had less than 10,000 instances ("Dinamina" and "Siyatha") we used the
original number of instances they contained. We divided this dataset into a
training and test set following a 0.8 split222The sampled dataset is available
at https://github.com/Sinhala-NLP/Sinhala-News-Media-Identification.

#### 3.1.2. Models

For this task, we experimented with several text classification models based
on transformers. From an input sentence, transformers compute a feature vector
$\bm{h}\in\mathbb{R}^{d}$, upon which we build a classifier for the task. For
this task, we implemented a softmax layer, i.e., the predicted probabilities
are $\bm{y}^{(B)}=softmax(W\bm{h}+b)$, where $W\in\mathbb{R}^{k\times d}$ is
the softmax weight matrix, and $k$ is the number of labels.

For these, we employed a batch-size of 16, Adam optimiser with learning rate
$2\mathrm{e}{-5}$, and a linear learning rate warm-up over 10% of the training
data. During the training process, the parameters of the transformer model, as
well as the parameters of the subsequent layers, were updated. The models were
evaluated at least three times per epoch while training using an evaluation
set that had one-fifth of the rows in training data. We performed early
stopping if the evaluation loss did not improve over three evaluation steps.
All the models were trained for three epochs.

#### 3.1.3. Results

We evaluate the models using the average weighted F1 score and macro F1 score.
The results are shown in Table 2.

Models | Weighted F1 | Macro F1
---|---|---
XLM-R Large | 0.8917 | 0.8763
XLM-R Base | 0.8961 | 0.8854
SinBERT Small | 0.8966 | 0.8883
SinBERT Large | 0.8967 | 0.8895

Table 2: News media classification results. The results are ordered with the
ascending order for Macro F1 score.

As can be seen in the results, the performance of all the transformer models
on the task was excellent. All the models achieved more than 0.88 Macro F1
score on the task. These outcomes imply that each news source exhibits a
distinct stylistic approach in presenting their news content.

Notably, the SinBERT Large model outperformed the XLM-R Large model. However,
the XLM-R Large model produces very close results to the SinBERT Large model.

### 3.2. News Category Prediction

The second task we compiled from NSina was news category prediction. This is
an old task in NLP, but it has many potential applications Bracewell et al.
(2009). Given the news content, the ML models should predict a pre-defined
category for the news.

#### 3.2.1. Data

Preparing the train/test sets for this task was more challenging than the
previous task as the categories are not commonly defined for all news media.
First, for this task, we dropped all the news articles without a category as
some news sources prefer not to categorise them. Next, we identified the top
100 news categories from the available news instances. We grouped them into
four main categories: local news, international news, sports news, and
business news. To avoid bias, we undersampled the dataset. We only used 10,000
instances from each category, and for the “Business" category, we used the
original number of instances which was 8,777 articles. We divided this dataset
into a training and test set following a 0.8 split333The sampled dataset is
available at https://github.com/Sinhala-NLP/Sinhala-News-Category-Prediction/.

#### 3.2.2. Models

We employed the same ML models experimented with for the previous task, as
this is also a text classification task. The hyperparameter configurations and
training strategy also remained the same.

#### 3.2.3. Results

Similar to the previous task, we used Weighted F1 and Macro F1 scores to
evaluate the models. The results are shown in Table 3.

Models | Weighted F1 | Macro F1
---|---|---
SinBERT Small | 0.9310 | 0.9306
SinBERT Large | 0.9370 | 0.9367
XLM-R Large | 0.9385 | 0.9381
XLM-R Base | 0.9387 | 0.9382

Table 3: News category prediction results. The results are ordered with the
ascending order for Macro F1 score. The best results are in bold.

Similar to the previous task, transformer models provided excellent results in
this task as well. This can be a particularly easy task as the news content
within each category we analysed is distinct and varied. Notably, XLM-R models
could outperform SinBERT model in this task.

### 3.3. News Headline Generation

Lastly, we created a natural language generation (NLG) task using the NSina
corpus, where the ML model’s objective is to generate news headlines based on
the provided news content. As all the news content in NSina has headlines,
constructing this NLG task was straightforward. In an era of increasing
interest in language generation models such as GPT, this benchmark will be
invaluable for evaluating their performance in the context of Sinhala language
generation.

#### 3.3.1. Data

We used the same instances from NSina as all the news articles had headlines.
We divided this dataset into a training and test set following a 0.8
split444The sampled dataset is available at https://github.com/Sinhala-
NLP/Sinhala-Headline-Generation.

#### 3.3.2. Models

In this task, we exploit two types of ML models based on transformers.

##### General Transformers

\- We created a Seq2Seq model from general transformers by adding a
transformer decoder, which takes the encoder’s output and generates the target
sequences. We experimented with several general-purpose transformer models
that support Sinhala, including XLM-Roberta Conneau et al. (2020), and SinBERT
Dhananjaya et al. (2022).

##### Text Generation Transformers

\- We also experimented with several text generation transformers as they have
provided excellent results in text generation tasks. Specifically, we explored
mBART Lewis et al. (2020) and several mT5 Xue et al. (2021) variants. Both
mBART and mT5 support Sinhala.

For both types of transformer models, we employed a batch size of 16, Adam
optimiser with learning rate $1\mathrm{e}{-4}$, and a linear learning rate
warm-up over 10% of the training data. During the training process, the
parameters of the transformer model were updated. The models were trained only
using the training data and evaluated while training using an evaluation set
that had one-fifth of the rows in training data. We performed early stopping
if the evaluation loss did not improve over three evaluation steps. All the
models were trained for three epochs.

#### 3.3.3. Results

We evaluated the results of the models using two popular NLG evaluation
metrics: BLEU Papineni et al. (2002) and Translation Edit Rate (TER). While
there are advanced NLG metrics such as BLEURT Sellam et al. (2020) and
BERTScore Zhang et al. (2019), they do not currently support Sinhala.

Models | BLEU | TER
---|---|---
SinBERT | 0.08 | 0.80
XLM-R Base | 0.11 | 0.76
XLM-R Large | 0.14 | 0.75
mBART | 0.15 | 0.74
mT5 Base | 0.16 | 0.73
mT5 Large | 0.17 | 0.72

Table 4: News category prediction results. The results are ordered in
ascending order for BLEU.

The results in Table 4 show that all the transformer models do not perform
well in this natural language processing task. The mT5-large model produced
the best result with a BLEU score of 0.17. However, a BLEU score in the range
of 0.1 and 0.2 suggests that there is a poor overlap between the generated
headline and the actual headline.

The poor performance on NLG can be attributed to two main reasons. (1) There
is no model trained specifically for Sinhala language generation and (2) The
BLEU and TER scores cannot properly evaluate Sinhala text generation and there
is a need for advanced NLG metrics for Sinhala.

## 4\. Conclusion

This research presented NSina, a large news corpus for Sinhala that can be
used to train LLMs. NSina is larger and more recent than previous news corpus
released for Sinhala, such as SinMin Upeksha et al. (2015). We also release
three benchmark datasets sampled from NSina that can be used to evaluate LLMs.
We evaluated several transformer models on each task. The results show that
multilingual transformer models such as XLM-R provide very close results or
sometimes even outperform language-specific models such as SinBERT Dhananjaya
et al. (2022), suggesting more research should be done to train Sinhala-
specific transformer models. Furthermore, all the experimented models perform
poorly on the proposed NLG task, suggesting that more language generation
models should be explored for Sinhala.

In future work, we would like to utilise NSina with other available Sinhala
resources to create robust transformer models. Furthermore, a GLUE Wang et al.
(2018) like benchmark will be created for Sinhala, including the tasks
proposed in this paper, which will serve as a pivotal platform for evaluating
the proficiency of LLMs in processing Sinhala.

## Acknowledgments

The computational experiments in this paper were conducted on the Aston EPS
Machine Learning Server, funded by the EPSRC Core Equipment Fund, Grant
EP/V036106/1.

## Ethics Statement

NSina was collected from publicly available websites, and none of the records
were edited in the process. Similar to the previous research that compiled and
released news corpora Nagoudi et al. (2020); Kakwani et al. (2020), for every
instance in NSina, we released the URL for the original news article.
Furthermore, we released NSina with the Creative Commons Attribution-Non
Commercial-ShareAlike 4.0 International Public License, which prevents users
from altering the instances in the dataset. While NSina is publicly available
in HuggingFace, we only released it as a gated dataset where the users need to
accept the license and request access. All the datasets released for the
subtasks in this paper also follow a similar process.

## 5\. Bibliographical References

##

  * Bracewell et al. (2009) David B Bracewell, Jiajun Yan, Fuji Ren, and Shingo Kuroiwa. 2009.  Category classification and topic discovery of japanese and english news articles.  _Electronic Notes in Theoretical Computer Science_ , 225:51–65.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * De Silva (2019) Nisansa De Silva. 2019.  Survey on publicly available sinhala natural language processing tools and research.  _arXiv preprint arXiv:1906.02358_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhananjaya et al. (2022) Vinura Dhananjaya, Piyumal Demotte, Surangika Ranathunga, and Sanath Jayasena. 2022\.  BERTifying Sinhala - a comprehensive analysis of pre-trained language models for Sinhala text classification.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 7377–7385, Marseille, France. European Language Resources Association.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin et al. (2022) Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022.  A survey of transformers.  _AI Open_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2020) Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020.  Extending multilingual BERT to low-resource languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2649–2656, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

## 6\. Language Resource References

##

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4344–4355, Marseille, France. European Language Resources Association.
  * Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020.  IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4948–4961, Online. Association for Computational Linguistics.
  * Nagoudi et al. (2020) El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020.  Machine generation and detection of Arabic manipulated and fake news.  In _Proceedings of the Fifth Arabic Natural Language Processing Workshop_ , pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Ranasinghe et al. (2024) Tharindu Ranasinghe, Isuri Anuradha, Damith Premasiri, Kanishka Silva, Hansi Hettiarachchi, Lasitha Uyangodage, and Marcos Zampieri. 2024.  SOLD: Sinhala offensive language dataset.  _Language Resources and Evaluation_.
  * Upeksha et al. (2015) Dimuthu Upeksha, Chamila Wijayarathna, Maduranga Siriwardena, Lahiru Lasandun, Chinthana Wimalasuriya, NHND De Silva, and Gihan Dias. 2015.  Implementing a corpus for sinhala language.  In _Symposium on Language Technology for South Asia 2015_.


# Sensitivity of Average Precision to Bounding Box Perturbations

Ali Borji
Quintic AI, San Francisco, CA
<EMAIL_ADDRESS>
###### Abstract

Object detection is a fundamental vision task. It has been highly researched
in academia and has been widely adopted in industry. Average Precision (AP) is
the standard score for evaluating object detectors. Our understanding of the
subtleties of this score, however, is limited. Here, we quantify the
sensitivity of AP to bounding box perturbations and show that AP is very
sensitive to small translations. Only one pixel shift is enough to drop the
mAP of a model by 8.4%. The mAP drop over small objects with only one pixel
shift is 23.1%. The corresponding numbers when ground-truth (GT) boxes are
used as predictions are 23% and 41.7%, respectively. These results explain why
achieving higher mAP becomes increasingly harder as models get better. We also
investigate the effect of box scaling on AP. Code and data is available at
https://github.com/aliborji/AP_Box_Perturbation.

##  1 Motivation

Tremendous success has been achieved in the area of object detection. Object
detection models have come a long way but performance is still low compared to
other vision tasks such as object recognition111The best score in terms of
mean AP (mAP) on COCO validation set is about 63.2%. Please see
https://paperswithcode.com/sota/object-detection-on-coco-minival.. The reasons
are still unknown [2, 3, 1]. We suspect that it may be partly due to how
models are evaluated, and more specifically, the way AP is computed. AP is a
complicated score and it is difficult to get all details right in implementing
this score [7]. Compared to huge efforts spent on building models, less
attention has been paid to understanding the scores for object detection.
Here, we study how errors in predicted boxes impact the accuracy.

Figure 1: Sensitivity of IOU to bounding box perturbation. Left) Proportional
shift, Right) Fixed pixel shift (here for two pixel shift).

Figure 2: Sensitivity of AP to bounding box perturbation.

##  2 Experiments and Results

We first study how shifting the bounding box impacts the IOU. Following from
that, we then study the effect of box translation on AP.

###  2.1 Sensitivity of IOU to bounding box translation

We randomly generated a number of bounding boxes with different widths and
heights. We then shifted the bounding boxes diagonally in two ways. In the
first approach, a box is shifted in steps proportional to the box size. Let
the tuple $(x_{l},y_{l},x_{r},y_{r})$ represent the original bounding box A.
The shifted box B is then computed as follows:

B = ($x_{l}$ \+ offset $\times$ W, $y_{l}$ \+ offset $\times$ H, $x_{r}$ \+
offset $\times$ W, $y_{r}$ \+ offset $\times$ H])

where W and H are width and height of the box, respectively. The offset varies
in the range $[0,1]$ in steps of 0.1222numpy.linspace(0,1,11).

In the second approach, we simply add a fixed offset to the box coordinates as
follows:

B = ($x_{l}$ \+ offset, $y_{l}$ \+ offset, $x_{r}$ \+ offset, $y_{r}$ \+
offset)

Here, offset varies for a number of pixels (_e.g._ 10).

The IOU of the boxes A and B over random boxes is shown in Fig. 1 for both
approaches. The higher the box shift, the lower the IOU. The drop in IOU is
not linear. The slope is sharper for small shifts than larger ones. As will be
shown later this also applies to the AP. The profile of decline in the first
approach is the same for all boxes regardless of the bounding box shape since
it is proportional to width and height. It does, however, depend on the
bounding box shape in the second approach (the right panel in Fig. 1). As
expected, small boxes are impacted much more than the larger ones for the same
amount of pixel shift.

###  2.2 Sensitivity of AP to bounding box translation

Here, we only consider box shifts in pixels since it is more intuitive and is
easier to interpret (_i.e._ approach two above). Eight directions for shifting
the box {left, right, top, down, top left, top right, bottom left, bottom
right} are included. The magnitude of shift (offset) varies in pixels from 0
to 10. Two regimes are considered. In the first regime, called random
direction perturbation, a bounding box is randomly shifted in one of the eight
directions. In the second regime, all boxes are shifted in the same direction.

Over the MS COCO validation set [5], we consider the ground truth bounding
boxes as predictions and give them score 1. We then compute the AP for these
predictions in both regimes. Results for the first regime are shown in Fig. 2.
As expected, increasing the shift lowers the performance across all three APs:
mAP, AP50, and AP75. The AP50 is less hindered compared to the other two APs
since a small shift does not change the IOU50 much. We observe that:

  * •

1 pixel shift of GT box (GT used as prediction) in a random direction lowers
the mAP about 23.8%, and 2 pixels shift lowers it about 41.6%,

  * •

1 pixel shift of GT box in a random direction lowers the AP50 about 3.8%, and
2 pixels shift lowers it about 13.1%,

  * •

1 pixel shift of GT box in a random direction lowers the AP75 about 18.7%, and
2 pixels shift lowers it about 30.6%,

  * •

1 pixel shift of GT box in a random direction lowers the mAP over large,
medium, and small objects 1.7%, 13.4%, and 41.7%, respectively.

With predictions of the MaskRCNN [4] (with their original scores), we find
that:

Figure 3: Sensitivity of AP to bounding box scaling.

  * •

1 pixel shift of model prediction in a random direction lowers the mAP about
8.4%, and 2 pixels shift lowers it about 20.1%,

  * •

1 pixel shift of model prediction in a random direction lowers the AP50 about
1.5%, and 2 pixels shift lowers it about 7.1%,

  * •

1 pixel shift of model prediction in a random direction lowers the AP75 about
9.6%, and 2 pixels shift lowers it about 25.4%,

  * •

1 pixel shift of model prediction in a random direction lowers the mAP over
large, medium, and small objects 1.38%, 5.85%, and 23.1%, respectively.

The threshold for an object to be considered small in COCO is area $32^{2}$.
Just one pixel shift in the box is enough to drop the mAP more than 40% using
GT boxes. Results for when boxes are all shifted in the same direction is
shown in Appendix for all directions. As expected diagonal directions hinder
AP more than horizontal or vertical directions. Drop in AP seems to be
symmetric (_e.g._ same amount for left and right directions).

###  2.3 Sensitivity to bounding box scaling

We also investigate two other transformations. In the first one, called
enlarging, box A is scaled according to the following formula:

B = ($x_{l}$, $y_{l}$, $x_{r}$ \+ offset, $y_{r}$ \+ offset)

Here, top left corner of the box is held fixed and the right corner is shifted
outwards. In the second type of scaling, called shrinking, the bounding box is
shrunk as follows:

B = ($x_{l}$, $y_{l}$, $x_{r}$ \- offset, $y_{r}$ \- offset)

As above, the offset varies from 0 to 10 pixels. Results are shown in Fig. 3.
It seems like both box transformations have similar effects. One pixel box
enlarging or shrinking lowers the mAP about 10% using GT bounding boxes (20%
over small objects) and 4% using model predictions (8% over small objects).
Scaling, the way we do here, hinders the AP less than box translations mainly
because the box is changed less.

##  3 Discussion and Conclusion

We find that AP is overly sensitive to bounding box precision. This may
explain why performance in object detection is saturating. Small errors in
predicted boxes are much more costly in higher APs than lower APs. For
example, improving from 0.9 to 0.91 mAP is much harder than improving from 0.4
to 0.41.

Our investigation suggests that we should seek other complementary ways to
evaluate object detectors. Some possibilities include a) designing alternative
scores (as in [6]), b) asking humans to judge how good a bounding box is, and
c) using detected boxes in downstream tasks (_e.g._ object classification).

The analyses of this sort can also give insights into the complexity of the
datasets (_i.e._ a measure of image or dataset clutter). For examples,
datasets for which AP drops quickly with box shifting may suggest that objects
are occluding each other a lot. Same analyses can also be done for 3D object
detection and other tasks such as object segmentation.

## References

  * [1] Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman.  Tide: A general toolbox for identifying object detection errors.  In European Conference on Computer Vision, pages 558–573. Springer, 2020.
  * [2] Ali Borji.  Empirical upper bound, error diagnosis and invariance analysis of modern object detectors.  arXiv preprint arXiv:2004.02877, 2020.
  * [3] Ali Borji and Seyed Mehdi Iranmanesh.  Empirical upper bound in object detection and more.  arXiv preprint arXiv:1911.12451, 2019.
  * [4] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [5] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [6] Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan.  Localization recall precision (lrp): A new performance metric for object detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 504–519, 2018.
  * [7] Rafael Padilla, Sergio L Netto, and Eduardo AB Da Silva.  A survey on performance metrics for object-detection algorithms.  In 2020 international conference on systems, signals and image processing (IWSSIP), pages 237–242. IEEE, 2020.

##  Appendix A Sensitivity of AP to bounding box perturbation

Figure 4: Sensitivity of AP to bounding box perturbation (left: Ground Truth
boxes as predictions, right: MaskRCNN predictions).

Figure 5: Sensitivity of AP to bounding box perturbation (continued).