# Enriching Word Usage Graphs with Cluster Definitions

###### Abstract

We present a dataset of word usage graphs (WUGs), where the existing WUGs for
multiple languages are enriched with cluster labels functioning as sense
definitions. They are generated from scratch by fine-tuned encoder-decoder
language models. The conducted human evaluation has shown that these
definitions match the existing clusters in WUGs better than the definitions
chosen from WordNet by two baseline systems. At the same time, the method is
straightforward to use and easy to extend to new languages. The resulting
enriched datasets can be extremely helpful for moving on to explainable
semantic change modeling.

Keywords: word usage graphs, semantic change detection, definition generation

Enriching Word Usage Graphs with Cluster Definitions

Mariia Fedorova1, Andrey Kutuzov1, Nikolay Arefyev1, Dominik Schlechtweg2
---
1University of Oslo, 2University of Stuttgart
{mariiaf, andreku<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction and related work

In natural languages most words are polysemous, i.e. the same word in
different contexts can have different meanings. This gave rise to NLP tasks
like word sense induction and word sense disambiguation. To solve them, the
NLP community came up with many valuable resources related to word senses. One
specific use case for such datasets is lexical semantic change modeling, where
diachronic changes of word meaning are traced in an automated way.

The field of semantic change modeling makes a heavy use of the so called ‘Word
Usage Graphs’ (WUGs) Schlechtweg et al. (2021). Each WUG is associated with a
particular target word, and is a weighted, undirected graph, with nodes
corresponding to target word usages in a corpus, while edges are weighted with
the semantic proximity of particular pairs of usages. WUGs are human-
annotated, with annotators yielding contextualized graded judgments about the
said semantic proximity. After the annotation is complete, graph nodes (target
word occurrences) are automatically clustered into groups roughly
corresponding to word senses. When a WUG contains target word occurrences from
different time periods, it is called a Diachronic Word Usage Graph (DWUG).

DWUGs are available for several languages and are often used for evaluating
semantic change detection and discovery systems Schlechtweg et al. (2020);
Kutuzov et al. (2022b). However, one of the features they lack is human
interpretability of senses: clusters are labeled only with their numerical
identifiers. To analyze a cluster from a DWUG, one has to browse through the
actual target word usages, which is cumbersome and time-consuming. Giulianelli
et al. (2023) addressed this issue and suggested a method to automatically
generate human-readable sense definitions (cluster labels) using an encoder-
decoder language model fine-tuned on the task of contextualized definition
generation Mickus et al. (2022). However, their work was limited to the
English DWUGs and they did not release the labels they produced.

In this paper, we apply this method to actually generate sense definitions for
the available diachronic and synchronic WUGs in several languages (English,
German, Norwegian, and Russian). We publicly release these enriched WUGs
(mappings from clusters to definitions) with definitions both in English and
in the WUG-specific
languages.111https://github.com/ltgoslo/wugs_with_definitions In addition, we
compare the performance of the LLM-based definition generation system with
other approaches which choose a definition from an existing ontology like
Wordnet or Wiktionary. Thus, our contribution is twofold: first, we make the
existing word usage graphs more useful for linguists and lexicographers;
second, we evaluate definition generation and aggregation methods on multiple
languages and release the best fine-tuned models.

## 2\. Data Description

### 2.1. Word usage graphs

The WUG repository Dominik Schlechtweg and Sabine Schulte im Walde (2023)
features word usage graphs for eight languages. We chose English, German,
Norwegian and Russian to experiment with. We ignore Chinese, Latin, Spanish
and Swedish for the time being: mostly because of the lack of available
datasets to fine-tune definition generators for these languages.

Language | Target words | Clusters | Clusters annotated | Diachronic?
---|---|---|---|---
English Schlechtweg et al. (2020) | 46 | 819 | 120 | True
German Schlechtweg et al. (2020) | 50 | 488 | 95 | True
Norwegian-1 Kutuzov et al. (2022a) | 40 | 99 | 23 | True
Norwegian-2 Kutuzov et al. (2022a) | 40 | 78 | 17 | True
Russian Aksenova et al. (2022) | 24 | 90 | 39 | False
Table 1: Main statistics of the enriched WUGs. We labeled and annotated only
clusters which feature at least three usage examples.

Table 1 provides the main statistics of the word usage graphs we employ. Note
that there are two Norwegian DWUGs with different target word sets and time
periods (Norwegian-1 and Norwegian-2). Also note that for Russian we use the
‘RuDSI’ WUGs Aksenova et al. (2022) which are not diachronic (target word
usages may come from different time period, but it is not specially accounted
for). There exist two semantic change detection datasets for Russian
(‘RuSemShift’ and ‘RuShiftEval’), but due to their annotation procedure, they
do not feature any meaningful clusters to label.

We did not produce labels for clusters with less than three example usages
(this excluded a large amount of singleton clusters). The reason is that many
of such ‘small’ groups are clustering errors, and even if they are not, it is
extremely difficult to come with a good definition based on one or two
examples; for the DefGen method (subsection 3.3), generating a definition for
a cluster with two examples is ill-defined. We leave labeling this long tail
of small clusters to future work.222Also, in some original DWUGs, there are
clusters labeled with ‘-1’, which means annotators were unsure about semantic
proximity for nodes within these clusters most of the time; we ignored these
clusters as well.

### 2.2. Definition datasets

To be able to generate contextualized dictionary-like definitions, a pre-
trained encoder-decoder language model has to be fine-tuned on a dataset of
definitions coupled to example target word usages. In addition, one has to
choose a prompt which serves as an instruction for the language model to do
the correct task. We follow the logic of Giulianelli et al. (2023) in choosing
both the prompt (‘What is the definition of TARGET_WORD?’ and its translations
into corresponding languages) and the fine-tuning datasets for English:
WordNet Ishiwatari et al. (2019), Oxford Gadetsky et al. (2018) and CoDWoE
Mickus et al. (2022) (all CoDWoE datasets originally come from Wiktionary).
For other languages we used the following resources:

  * •

Russian: the corresponding part of CoDWoE;

  * •

Norwegian: Bokmålsordboka The Norwegian Language Council and the University of
Bergen (2023).

We are not aware of any readily available definition dataset for German;
however, we still decided to assign cluster labels to German DWUGs as an
experiment in zero-shot cross-lingual definition generation (also, we had
available resources for human evaluation of German cluster labels). See 3 for
more details on definition generation for different languages.

Table 2 shows the statistics of the definition datasets for fine-tuning.

Dataset | Entries | Lemmas | Ratio | Usage Length | Definition Length
---|---|---|---|---|---
English | 175 332 | 49 238 | 3.56 | $31.64^{\pm 23.82}$ | $17.56^{\pm 11.13}$
Norwegian | 70 711 | 30 551 | 2.31 | $8.69^{\pm 3.40}$ | $12.25^{\pm 7.35}$
Russian | 72 872 | 34 495 | 2.11 | $52.03^{\pm 29.37}$ | $24.65^{\pm 15.84}$
Table 2: Definition datasets. Average usage and definition lengths are given
in mT0 sub-words.

## 3\. Definition generators

We present three methods that can enrich DWUG clusters with definitions. Two
of them are our baselines: they select a definition for each cluster from a
human-curated lexical ontology, the English WordNet in our case. The third one
generates definitions from scratch and is our main method.

### 3.1. Lesk

The simplest method uses the Lesk algorithm Lesk (1986) originally developed
to solve the Word Sense Disambiguation (WSD) task. For a target word in a
given context, it browses all the definitions listed for this word in a
glossary or some WordNet-like resource and picks up a definition with the
highest lexical overlap with the given context. To construct a context for our
task we concatenate all usages from the same cluster, thus selecting the most
suitable definition for this particular cluster. The NLTK Bird et al. (2009)
implementation of the Lesk algorithm was employed, together with WordNet as
the source of senses and their definitions. We used the pre-tokenized versions
of WUG examples, while WordNet definitions were split into tokens by
whitespaces. The part-of-speech tags of the target words (where available)
were taken into account when selecting synsets. This method is doomed to fail
when the target word occurs in a sense that is not listed among senses of this
target word in WordNet, or if the target word is simply absent in WordNet.
Also, it does not work with languages other than English unless using lexical
resources for those languages. In our experiments with the English DWUGs, we
observed very low accuracy compared to other methods even for English which
has well-developed lexical resources (see Section 4). Thus, we decided not to
adapt Lesk to other languages.

### 3.2. GlossReader

GlossReader (Rachinskiy and Arefyev, 2021) is a system originally developed to
produce the contextualized embeddings that were shown to outperform standard
LLM embeddings in lexical semantic tasks such as Multilingual and Cross-
lingual Word-in-Context (Martelli et al., 2021) and Lexical Semantic Change
Discovery (Zamora-Reina et al., 2022). The system is based on the English WSD
model by Blevins and Zettlemoyer (2020), which consists of a definition
(gloss) encoder and a context encoder, both initialized with the English BERT
weights and jointly fine-tuned on a WSD dataset to maximize the dot product
between a context embedding and a definition embedding of the corresponding
word sense. In GlossReader, the English BERT backbone was replaced by the
multilingual XLM-R Conneau et al. (2020) in both encoders, and it was shown
that fine-tuning XLM-R as part of such WSD system significantly improves the
performance of its contextualized embeddings for several lexical semantic
tasks in various languages even when fine-tuning on the same English WSD
dataset only (Rachinskiy and Arefyev, 2021, 2022). In this work, we first
build gloss embeddings for all WordNet definitions (about 117K) with the gloss
encoder. Then, for each word usage we build the contextualized embedding of
the target word and retrieve $k$ most similar definition embeddings333The
pairwise human annotations from DWUGs were employed to select the optimal
value of $k$ among 1,3,10. For each DWUG and each $k$ we built a definition
for each cluster, then assigned cluster definitions to all of its usages and
estimated the probability that two usages with entirely different senses (the
annotation of 1) obtain the same definition. We selected $k=3$ for English
DWUG and $k=10$ for other DWUGs to minimize this probability. as measured by
the dot product. Finally, for each cluster we select the definition retrieved
for the largest number of usages in this cluster.

Unlike Lesk, this method selects from all definitions in WordNet, thus, it can
produce reasonable definitions even for senses not listed for the target word
in WordNet. Moreover, due to zero-shot cross-lingual transferability of the
XLM-R backbone, GlossReader can produce reasonable embeddings for definitions
and word usages in various languages, and retrieve definitions in one language
for word usages in another language. In this work, we use GlossReader to
retrieve definitions in English for word usages in several other languages
(cross-lingual setup), but we leave experiments with retrieving definitions in
other languages (multilingual setup) for the future work.

### 3.3. DefGen

We dub ‘DefGen’ our main system, which takes as an input a text string
containing a target word usage, and generates a human-readable definition of
the target word in this particular context from scratch. We use the method
proposed by Giulianelli et al. (2023): an encoder-decoder language model is
fine-tuned on a dataset of target word usages and the corresponding
definitions (see the subsection 2.2). Then, definitions are conditionally
generated for every example in the test set (in our case, WUGs). Giulianelli
et al. (2023) used Flan-T5 Chung et al. (2022) as the underlying language
model. However, it was trained predominantly on English and lacks the
capability to properly encode or generate texts in languages with
significantly different writing systems (especially true for Russian, but some
German and Norwegian characters are also not processed by the Flan-T5
tokenizer). Because of that, in this work we are using mT0 Muennighoff et al.
(2023) which is essentially a multilingual version of Flan-T5 (also fine-tuned
on many datasets cast as natural language instructions). For all the
experiments, we employ the mT0-xl version of the
model444https://huggingface.co/bigscience/mT0-xl, 3.7B parameters in size.

Fine-tuning was done in a standard text-to-text setup, for every language
(English, Norwegian, Russian) separately, so that in the end we had three
language-specific models. However, the underlying model is multilingual, so
e.g., the English DefGen can still take sentences in other languages as input,
and produce target word definitions in English with the standard English
prompt. We used it to generate English definitions for the German DWUGs. Here,
the input to the model looked like ‘Ist eine Prüfung erforderlich, so erfolgt
eine Entscheidung über den Antrag durch die zuständige Behörde. What is the
definition of Entscheidung?’ (‘If an examination is necessary, a decision on
the application will be made by the responsible authority. What is the
definition of decision?’)555The mT0 answer in this case was ‘The act of making
up your mind about something; a decision.’. Note that the models were not
specifically fine-tuned on producing English definitions for German examples:
this is a zero-shot multilingual ability probably stemming from the large-
scale instruction fine-tuning of the base model on other tasks.

Table 3 shows the raw performance of the fine-tuned models with English and
language-specific (‘native’) prompts as ROUGE-L Lin (2004) on the validation
set. Note that here we report only the performance for definitions generated
in the same language as the original examples (we don’t have any datasets with
examples in one language and definitions in another to evaluate against).
ROUGE-L measures only surface form overlap, so it should not be considered as
the definitive metric. Still, the performance of our definition generators for
English and Norwegian is on par with the English results reported in
Giulianelli et al. (2023). The lower score on the Russian CodWoE dataset is
most probably related to the abundance of dictionary-specific abbreviations,
to morphological richness of Russian and to the fact that the reference
Russian definitions are longer on average than those for other languages, as
well as usage examples (Table 2). This is why the lengths of the longest
common sub-sequence present in the reference definition and the generated
definition are lower on average. Still, the definitions generated by the
Russian model are mostly relevant and meaningful, as confirmed with visual
examination by native Russian speakers and with the human evaluation
experiments in Section 4. It is also worth noting that the performance when
using English or language-specific prompts is not significantly different. In
all the subsequent experiments, we use native prompts when generating
language-specific definitions.

Language | English prompt | Native prompt
---|---|---
English | 39.14 | 39.14
Norwegian | 28.16 | 27.76
Russian | 17.26 | 17.25
Table 3: Performance of mT0-based definition generators (ROUGE-L * 100) on the
validation sets.

Once the definitions for all the examples of a specific WUG cluster in the
desired language are generated, we again follow Giulianelli et al. (2023) in
choosing the most prototypical definition as the cluster label. All the
generated definitions are encoded by a multilingual SBERT
model666https://huggingface.co/sentence-transformers/distiluse-base-
multilingual-cased-v1 Reimers and Gurevych (2019) and the most prototypical
definition is trivially the one with the vector representation closest (by
cosine similarity) to the average of all the definition embeddings in the
cluster. No search for hyperparameters was conducted, we used the default
values from Giulianelli et al. (2023) at all the DefGen stages.

In this way, we generated English cluster/sense labels for all four languages.
In addition, language-specific labels for Norwegian and Russian were
generated. In the next section, we evaluate them using human judgments.

## 4\. Evaluation and results

### 4.1. Evaluation setup

Our ultimate aim is to assign each WUG cluster a human-readable label which is
distinct enough to be useful for a linguist or lexicographer. In other words,
the label should be helpful in distinguishing one cluster from another777The
clusters themselves can be incorrect, but their fixing is out of scope for
this paper.. To evaluate the labeling methods, we came up with the ‘guess the
cluster by definition’ task for human evaluation, described below.

For every cluster of a target word, the annotators were shown the label
generated for this cluster (without knowing what system it came from) and two
clusters, represented with five randomly sampled example sentences each (or
less, if the cluster has less examples in the WUG). One of those clusters is
the one for which the system generated the label and another one is a randomly
sampled filler cluster (the filler pairings were generated before the
evaluation started and were used throughout the whole process). The humans
were asked which cluster is the best fit for the shown label (definition).
They also had the options ‘none of the clusters fits’ and ‘the definition fits
both clusters’. Only one choice was allowed. Importantly, the annotators were
instructed that the label fits the cluster if it fits the majority of the
examples in it. Otherwise, fluency or factual correctness of the definitions
were not taken into account: they only had to be distinctive enough to tell
one cluster from another. Our final evaluation metrics is accuracy for all the
clusters, where the system label is considered to be ‘correct’ if and only if
the human chose as the best fit the same cluster that the system generated the
label for. Our additional metrics are the percentages of ‘fits both’ and ‘fits
none’ judgments (the less the better).

German, Russian and Norwegian predictions were evaluated by native speakers of
the corresponding languages (the German speaker was familiar with the
corresponding WUG data beforehand). English predictions were evaluated by the
paper authors who are fluent English speakers. English definitions for the
RuDSI dataset were evaluated by three independent annotators (using majority
voting as the aggregation method). The resulting Krippendorff’s $\alpha$
inter-rater agreement is 0.314, which is considered a fair agreement Artstein
and Poesio (2008).

### 4.2. Experimental results

Table 4 shows the evaluation results of the systems for the WUGs we experiment
with. Note that the task we are solving (generating labels or definitions for
sets of sentences) is novel in itself, so there are no prior results to
compare against. DefGen outperforms both Lesk and GlossReader in accuracy for
all the datasets under comparison (English, German and Russian with English
definitions). Interestingly, DefGen is slightly more prone to producing too
general definitions which fit both clusters, but much more rarely produces
irrelevant definitions which fit none of the clusters (for Lesk, more than
half of the definitions are like this). This is probably the reason for the
higher DefGen accuracy score.

For the rest of the WUGs (Norwegian with Norwegian and English definitions and
RuDSI with Russian definitions), we evaluated only DefGen, to make sure its
performance does not drop for some reason. The results for Norwegian are on
the same level or higher, but the accuracy of Russian definitions for RuDSI
are much lower than the English definitions generated with the same method.
This partially stems from an increased number of ‘fits none’ judgments (the
definition is irrelevant for both clusters). We observed low ROUGE-L
performance of the Russian DefGen before (subsection 3.3), so it does not come
as a surprise.

However, this degradation can be caused by some properties of a particular
dataset, as is suggested by the difference between English and Norwegian
definitions for the Norwegian clusters. As mentioned before, there are two
sets of Norwegian DWUGs. For Norwegian-2, the English definitions are more
accurate than the Norwegian ones (88.24 vs 76.47), same tendency as for
Russian. But for Norwegian-1, it’s vice versa (60.87 vs 73.91). Thus, the
performance of English definitions for Norwegian examples varies greatly: can
be excellent for one set of WUGs but poor for another. At the same time, the
performance of Norwegian definitions is fairly good for both. We conjecture
that the definitions in WUG-native languages can be more robust and less
influenced by peculiarities of specific target words. If this is true, it
would mean that RuDSI simply happened to be a ‘lucky’ set of WUGs for English
definitions, same as Norwegian-2. To test this hypothesis, another Russian
dataset is needed, so this is left for future work.

System | Accuracy | Fits both | Fits none
---|---|---|---
English DWUG, English definitions
Lesk | 21.67 | 5.00% | 53.33%
GlossReader | 50.00 | 9.17% | 37.50%
DefGen | 69.17 | 10.83% | 11.67%
German DWUG, English definitions
GlossReader | 53.68 | 13.68% | 27.37%
DefGen | 57.89 | 16.84% | 12.63%
RuDSI, English definitions
GlossReader | 64.10 | 10.26% | 17.95%
DefGen | 71.79 | 15.38% | 2.56%
Norwegian-1 DWUG, English definitions
DefGen | 60.87 | 13.04 | 21.74
Norwegian-2 DWUG, English definitions
DefGen | 88.24 | 5.88% | 5.88%
Norwegian-1 DWUG, Norwegian definitions
DefGen | 73.91 | 4.35% | 21.74%
Norwegian-2 DWUG, Norwegian definitions
DefGen | 76.47 | 11.76% | 11.76%
RuDSI, Russian definitions
DefGen | 48.72 | 7.69% | 15.38%
Table 4: Results of human evaluation in the ‘guess the cluster by definition’
task.

### 4.3. Error analysis

We have manually classified all erroneous definitions from the English,
Norwegian 1 and Russian DWUG by mistake type, the results are shown in the
tables 5, 6, 7.

Error type Share Number too broad definition 0.41 15 wrong sense 0.32 12
similar words but not a definition 0.16 6 redundant whitespace 0.08 3
ambiguous definition, ambiguous word 0.03 1 repetitions 0.05 2 factual mistake
0.03 1 non-existing word 0.03 1 Total 1.00 37

Table 5: Erroneous English definitions by error type. Some definitions were
annotated with two error types, for instance, some too broad definitions also
suffer from a redundant whitespace.

One of the most widespread problems, which has caused most of ‘fits both’
annotations, is too broad definition. For example, the following definition
was generated for the word ‘relationship’: ‘The way in which two or more
things are connected, or in which two or more things are related.’. This
definition did not allow the annotator to distinguish the clusters containing
usages related to relationship among entities or among people. The same word
causes ‘fits both’ in Norwegian: ‘forhold’ (‘relationship’) is defined as
‘forbindelse, samsvar mellom ulike faktorer’ (‘connection, agreement between
different factors’). A similar problem was also reported by the annotator of
the German data: ‘Concreteness vs. abstractness seems very important in
descriptions: Sometimes the description could fit to both clusters, depending
on whether it is read concretely or abstractly (metaphorically), e.g.
‘beimischen’, ‘to mix (something) together’. Mixing could be done e.g. by
mixing chemicals or ideas or thoughts. Similarly with ‘abdecken: to cover or
supply’.

Error type Share Number wrong sense 0.5 3 similar words but not a definition
0.17 1 too broad definition 0.17 1 repetitions 0.17 1 wrong preprocessing 0.17
1 Total 1.00 6

Table 6: Erroneous Norwegian definitions by error type. Some definitions were
annotated with two error types.

Another source of too broad definitions is joining multiple meanings of a word
into one definition by semicolon (there can be also a redundant space added
before this semicolon; we consider such cases to be a punctuation error. A
possible reason is that the training data were pre-tokenized and joined by a
space). For example, ‘plane’ is defined as ‘A flat surface, without slope,
tilts, or indentations ; a level surface.’ which merges its everyday and
mathematical meanings.

Some definitions use ambiguous words so that the resulting sense description
does not help to distinguish between clusters. For example, ‘Something used to
support or maintain an object, scene, etc.’ for the word ‘prop’. The word
‘scene’ is also ambiguous and the annotator has selected a wrong cluster
because of it. This problem arises even more clearly when the model is trained
in other languages with an English prompt. The definitions are often simply
direct translations of the target word into English with the same level of
ambiguity. For example, the Russian word ‘ сторона’ (‘side’) is defined as
‘one’s part, aspect, role’.

Some of ‘fits both’ annotations may be also explained by too granular
clustering, but fixing it is beyond of the scope of this paper.

Another frequent problem are definitions that are good themselves, but
describe another sense of the word, not the one used in their source clusters.
Such definitions result in wrong or ‘fits none’ annotation. They are often
generated by usages of the target lemma in phrasal verbs or in metaphorical
meaning. For example, a cluster containing usages of ‘chef d’ouevres’ (in the
sense of ‘masterpiece’) has generated the definition ‘A master of excellence.’
for the target word ‘chef’.

Error type Share Number wrong sense 0.5 10 repetitions 0.45 9 similar words
but not a definition 0.3 6 redundant whitespace 0.2 4 redundant note 0.2 4 too
broad definition 0.15 3 redundant parentheses 0.05 1 Total 1.00 20

Table 7: Erroneous Russian definitions by error type. Some definitions were
annotated with more than one error type.

The third most frequent problem are definitions that are semantically similar
to what could be a good definition, but they barely describe the meaning of
the cluster. Such cases are called ‘similar words but not a definition’ in our
tables. An example is the definition of the word ‘gas’: ‘Any fluid substance,
especially a natural fluid, which is a mixture of air and nitrogen, usually
created by the combustion of natural gasses.’ It sounds much dictionary-like,
but is meaningless and also contain a non-existing word ‘gasses’. The same
problem was also reported for German, the annotator has often made his
judgments based on ‘only some aspect that was not present in the other
cluster’ in such cases. For Norwegian, there is a definition ‘liten,
sammenhengende fjordformasjon’ (‘small, connected fjord formation’) generated
for the word ‘rev’ (‘reef’). Not only is this definition an example of over-
fitting to the Norwegian realities, but also, as the annotator has pointed
out, ‘fjordformasjon’ means ‘a formation of the actual fjord, like how the
water moves and the sides of the fjord, not the ground in the water in the
fjord’. Sometimes the meaninglessness of the definition is produced by
repetitions in it; for example, ‘pin’ is defined as ‘To fasten with a pins or
pins.’

Sometimes the definitions with a wrong meaning still carry the sentiment or
the style of their source cluster. For example, the definition ‘Bravery,
courage, chutzpah, or brazenness.’ containing a colloquial word ‘chutzpah’ was
generated for the word ‘ball’ in its colloquial meaning. For German, the
annotator could assign the definition to one of the clusters only based on the
sentiment while the definition content does not fit otherwise: ‘Abgesang: an
expression of disapproval, a denial, or rejection. The negative sentiment fits
very well with the "downfall" meaning of the word’.

Factual mistakes are an important problem. For example, ‘heel’ is defined as
‘The part of the human foot above the ankle.’, while it is the part under the
ankle. The annotation
guidelines888https://github.com/ltgoslo/wugs_with_definitions/blob/main/human_evaluation/evaluation_guidelines.pdf
included the instruction to ignore factual mistakes if they allow to
distinguish between the clusters, so in fact this type may be not as rare as
our current analysis shows. For German, the annotator reported that ‘the
meaning expressed an antonym or contrast of one of the clusters: e.g.
‘Schmiere: a highly successful theatrical production’ is exactly the opposite:
a very bad theatrical play. I judged it still as fitting. Similarly for:
Engpass: the property of a more than adequate quantity or supply; Spielball:
invloving active participation; Missklang: any agreeable (pleasing and
harmonious) sounds’.

Some errors are specific for the Russian training dataset and can be fixed by
cleaning the dataset. These are the Wiktionary artifacts: redundant footnotes
in parentheses (e.g. ‘стороны [ 1 ], участники какого-либо конфликтного
процессуального действия’) and redundant dictionary notes that seem to occur
randomly, e.g. ‘только полн. ф.’ (‘full form only’), which makes sense for
adjectives and participles only, but is generated in the definition of a noun.

To conclude, the definitions produced by the model are not yet ready-to-use.
They are in general suitable for semantic change detection or other semantic-
related NLP tasks, but not for the full replacement of a human lexicographer.

More definition examples with an attempt to classify them by the error
severity into good, borderline and bad ones can be found in Appendix A.

## 5\. Conclusion

We have experimented with several methods for assigning human-readable cluster
labels to word usage graphs (both synchronic and diachronic) for multiple
languages. Human evaluation showed that conditional generation of such labels
from scratch using a fine-tuned mT0 language model outperforms two baselines
which choose a label from an external closed set of possible definitions
(senses). This ‘DefGen’ method generally follows the prior work, but we
additionally show that it can generate reasonable labels even for languages on
which the model was not specifically fine-tuned. Our definition generation
models for English999https://huggingface.co/ltg/mt0-definition-en-xl,
Norwegian101010https://huggingface.co/ltg/mt0-definition-no-xl and
Russian111111https://huggingface.co/ltg/mt0-definition-ru-xl are publicly
available.

We release enriched versions of the existing WUGs for English, German, Russian
and Norwegian. In these versions, clusters of usages are accompanied with
labels generated by DefGen, both in English and in the native languages of the
datasets. It is our hope that this makes the WUGs easier to use and analyze
for linguists and lexicographers: it is now not necessary to browse through
all the examples in a cluster to find out what sense of the target word does
it correspond to. It is even more important for diachronic WUGs, where senses
are changing over time and must be compared historically. Assigning DWUG
clusters human-readable definitions is another step towards explainable
semantic change modeling, thus bridging the gap between NLP and humanities.

We believe our findings support the claim of Giulianelli et al. (2023) that
generated definitions can be used as convenient and interpretable
contextualized representations for a wide range of domains. One can think of
using such ‘definitions as representations’ in WSD, WSI or sentiment analysis,
to name only a few NLP tasks.

As a future work, we plan to experiment with a DefGen model fine-tuned on
definition datasets in several languages at once and with further fine-tuning
such a model on a very small amount of manually created examples in other
languages. Our preliminary experiments with German are promising (the model is
able to generate definitions in German, despite the lack of proper fine-tuning
dataset), but more work is needed. It is also planned to generate cluster
labels for other existing WUGs not yet covered in this paper.

## 6\. Acknowledgments

Andrey Kutuzov and Nikolay Arefyev have received funding from the European
Union’s Horizon Europe research and innovation program under Grant agreement
No 101070350 (HPLT). Dominik Schlechtweg has been funded by the research
program ‘Change is Key!’ supported by Riksbankens Jubileumsfond (under
reference number M21-0021). We acknowledge the help of Petter Mæhlum in
evaluating Norwegian definitions and Pavel Suvorkov in evaluating Russian
definitions.

## 7\. Bibliographical References

## References

  * Aksenova et al. (2022) Anna Aksenova, Ekaterina Gavrishina, Elisei Rykov, and Andrey Kutuzov. 2022.  RuDSI: Graph-based word sense induction dataset for Russian.  In _Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing_ , pages 77–88, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Artstein and Poesio (2008) Ron Artstein and Massimo Poesio. 2008.  Survey article: Inter-coder agreement for computational linguistics.  _Computational Linguistics_ , 34(4):555–596.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Blevins and Zettlemoyer (2020) Terra Blevins and Luke Zettlemoyer. 2020.  Moving down the long tail of word sense disambiguation with gloss informed bi-encoders.  In _Proceedings of the 58th Association for Computational Linguistics_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Gadetsky et al. (2018) Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018.  Conditional generators of words definitions.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 266–271, Melbourne, Australia. Association for Computational Linguistics.
  * Giulianelli et al. (2023) Mario Giulianelli, Iris Luden, Raquel Fernandez, and Andrey Kutuzov. 2023.  Interpretable word sense representations via definition generation: The case of semantic change analysis.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3130–3148, Toronto, Canada. Association for Computational Linguistics.
  * Ishiwatari et al. (2019) Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019.  Learning to describe unknown phrases with local and global contexts.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3467–3476, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kutuzov et al. (2022a) Andrey Kutuzov, Samia Touileb, Petter Mæhlum, Tita Enstad, and Alexandra Wittemann. 2022a.  NorDiaChange: Diachronic semantic change dataset for Norwegian.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2563–2572, Marseille, France. European Language Resources Association.
  * Kutuzov et al. (2022b) Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2022b.  Contextualized embeddings for semantic change detection: Lessons learned.  In _Northern European Journal of Language Technology, Volume 8_ , Copenhagen, Denmark. Northern European Association of Language Technology.
  * Lesk (1986) Michael Lesk. 1986.  Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone.  In _Proceedings of the 5th annual international conference on Systems documentation_ , pages 24–26.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Martelli et al. (2021) Federico Martelli, Najla Kalach, Gabriele Tola, and Roberto Navigli. 2021.  SemEval-2021 task 2: Multilingual and cross-lingual word-in-context disambiguation (MCL-WiC).  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 24–36, Online. Association for Computational Linguistics.
  * Mickus et al. (2022) Timothee Mickus, Kees Van Deemter, Mathieu Constant, and Denis Paperno. 2022.  Semeval-2022 task 1: CODWOE – comparing dictionaries and word embeddings.  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_ , pages 1–14, Seattle, United States. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023\.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Rachinskiy and Arefyev (2021) Maxim Rachinskiy and Nikolay Arefyev. 2021.  GlossReader at SemEval-2021 task 2: Reading definitions improves contextualized word embeddings.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 756–762, Online. Association for Computational Linguistics.
  * Rachinskiy and Arefyev (2022) Maxim Rachinskiy and Nikolay Arefyev. 2022.  GlossReader at LSCDiscovery: Train to select a proper gloss in English – discover lexical semantic change in Spanish.  In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_ , pages 198–203, Dublin, Ireland. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Schlechtweg et al. (2020) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020.  SemEval-2020 task 1: Unsupervised lexical semantic change detection.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 1–23, Barcelona (online). International Committee for Computational Linguistics.
  * Schlechtweg et al. (2021) Dominik Schlechtweg, Nina Tahmasebi, Simon Hengchen, Haim Dubossarsky, and Barbara McGillivray. 2021.  DWUG: A large resource of diachronic word usage graphs in four languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7079–7091, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zamora-Reina et al. (2022) Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022.  LSCDiscovery: A shared task on semantic change discovery and detection in Spanish.  In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_ , pages 149–164, Dublin, Ireland. Association for Computational Linguistics.

## 8\. Language Resource References

##

  * Dominik Schlechtweg and Sabine Schulte im Walde (2023) Dominik Schlechtweg and Sabine Schulte im Walde. 2023.  _Word Usage Graphs (WUGs)_.  University of Stuttgart.
  * The Norwegian Language Council and the University of Bergen (2023) The Norwegian Language Council and the University of Bergen. 2023.  _Bokmålsordboka_.  University of Bergen.

##  Appendix A Appendix

Figure 1: Diachronic word usage graph of the Norwegian word ‘rev’ (‘fox’). The
blue cluster contains the usages with the sense ‘fox’, the green cluster the
usages with the sense ‘strip of sail’, the orange one the usages with the
sense ‘reef’ and the rose cluster one usage with the sense ‘card play’. It can
be seen that the sense ‘card play’ no longer exists in the second time period,
and the sense ‘strip of sail’ has become rare.

word DWUG sense prompt definition translation Entscheidung German decision
What is the definition of Entscheidung? The act of making up your mind about
something; a decision. мир Russian peace Что такое мир? полит. соглашение
между сторонами о взаимных отношениях in politics: agreement between two sides
about their mutual relationships

Table 8: Good definition examples

word DWUG sense prompt definition translation error rev Norwegian-1 fox Hva
betyr rev? rovdyr som blir brukt som inntektskilde predator that is used as a
source of income too broad definition rev Norwegian-1 reef What is the
definition of rev? A reefs, especially coral reef walls. grammar mistake мир
Russian community of people Что такое мир? перен. совокупность людей,
объединённых общностью интересов, интересов, а также их общая совокупность
metaphorical: a community of people with shared interests, interests, and also
their entirety repetitions plane English What is the definition of plane? An
aircraft. too broad definition

Table 9: Borderline definition examples

word DWUG sense prompt definition translation error rev Norwegian-1 fox What
is the definition of rev? A rabbit. factual mistake rev Norwegian-1 strip of
sail Hva betyr rev? særlig skrevet særlig skrevet … -, meaningless repetition
of random words wrong preprocessing or bad generation parameters rev
Norwegian-1 strip of sail What is the definition of rev? A twisting blow ; a
punch. similar words but not a definition

Table 10: Bad definition examples


# Errors When Constraining Hot Blackbody Parameters with Optical Photometry

Iair Arcavi The School of Physics and Astronomy, Tel Aviv University, Tel Aviv
69978, Israel  CIFAR Azrieli Global Scholars program, CIFAR, Toronto, Canada
Iair Arcavi<EMAIL_ADDRESS>
###### Abstract

Measuring blackbody parameters for objects hotter than a few $10^{4}$ K with
optical data alone is common in many astrophysical studies. However this
process is prone to large errors because at those temperatures the optical
bands are mostly sampling the Rayleigh–Jeans tail of the spectrum. Here we
quantify these errors by simulating different blackbodies, sampling them in
various bands with realistic measurement errors, and re-fitting them to
blackbodies using two different methods and two different priors. We find that
when using only optical data, log-uniform priors perform better than uniform
priors. Still, measured temperatures of blackbodies above $\sim$35,000 K can
be wrong by $\sim$10,000 K, and only lower limits can be obtained for
temperatures of blackbodies hotter than $\sim$50,000 K. Bolometric
luminosities estimated from optical-only blackbody fits can be wrong by
factors of 3–5. When adding space-based ultraviolet data, these errors shrink
significantly. For when such data are not available, we provide plots and
tables of the distributions of true temperatures that can result in various
measured temperatures. It is important to take these distributions into
account as systematic uncertainties when fitting hot blackbodies with optical
data alone.

High energy astrophysics (739), Astronomical methods (1043), Optical astronomy
(1776), Ultraviolet astronomy (1736)

††software: PySynphot (STScI Development Team, 2013), Lightcurve_Fitting
(Hosseinzadeh & Gomez, 2021), emcee (Foreman-Mackey et al., 2013).

##  1 Introduction

In many astrophysical studies it useful to fit a blackbody spectrum to broad-
band photometry in order to determine the effective temperature, emitting
radius, and bolometric luminosity of an object or transient event. Hot
blackbodies (at temperatures of a few $10^{4}$ K) are of particular interest,
for example in certain tidal disruption events (see van Velzen et al. 2020 and
Gezari 2021 for recent reviews), in the early phases of supernovae (e.g.
Valenti et al., 2016) and in the very early phases of kilonovae (at least
based on the one case observed so far; Abbott et al., 2017).

Constraining the temperature, and with it the bolometric luminosity, of tidal
disruption events is important for distinguishing between emission models, and
for deriving properties of the supermassive black hole population (e.g. Piran
et al., 2015; Dai et al., 2018; Mockler et al., 2019; Ryu et al., 2020).
Measuring the cooling rate of supernovae during their first days can be used
to constrain both progenitor parameters and explosion physics (e.g. Nakar &
Sari, 2010; Rabinak & Waxman, 2011; Shussman et al., 2016; Arcavi et al.,
2017; Rubin & Gal-Yam, 2017; Sapir & Waxman, 2017). For kilonovae to be used
to constrain the neutron-star equation of state, heavy-element
nucleosynthesis, and even cosmology, it turns out that measuring their cooling
during the first few hours is especially important (e.g. Arcavi, 2018).

However, in many of these cases (when not highly redshifted), the optical
wavelength regime is in the Rayleigh–Jeans tail of the spectrum, where it is
difficult to constrain the temperature with optical data alone. Figure 1
illustrates how spectra of blackbodies hotter than approximately 30,000 K are
very similar in the optical bands. It is therefore expected that optical data
alone will not be able to clearly distinguish between blackbodies at those
temperatures, especially when measurement uncertainties are taken into
account.

Figure 1: Blackbody spectra at various temperatures (denoted in kilo-Kelvin),
normalized to their flux at 6,000 Å. Swift $uvw1$ (abbreviated as $w1$),
$uvm2$ ($m2$) and $uvw2$ ($w2$) filter response curves are shown at the bottom
together with those of the optical-infrared $ugriz$ (solid lines) and $UBVRI$
(dotted lines) bands. Blackbodies at temperatures $\gtrsim$30,000 K all appear
very similar in the optical to near-infrared regime.

Indeed, Faran et al. (2018) show that when approaching temperatures of 20,000
K, temperature estimates based on optical data alone can be off by
$\gtrsim$1,000 K. Here we expand the quantification of such measurement errors
and characterize them also at higher temperatures. We simulate observations of
blackbodies at various temperatures, taking into account realistic measurement
uncertainties, in both optical and ultraviolet bands, and fit the data to
blackbodies as would be done for real observations (Section 2). We then
measure how different the best-fit temperature is from the true simulated
temperature for different temperatures, band combinations, methods, priors,
and measurement uncertainties (Section 3). These results are then inverted in
order to determine, given an optical-only temperature measurement, what true
temperature values could produce it. We also check what effect temperature
measurement errors have on errors in the deduced bolometric luminosity
(Section 4).

We do not consider real-life complications such as K-corrections, distance
uncertainties, extinction uncertainties, and line emission and absorption
contamination and blanketing. Extinction and line blanketing are especially
important to consider with ultraviolet data, however here we wish to isolate
the degeneracies and systematic errors of fitting just a hot blackbody
component with optical data, and compare it to using also ultraviolet data.

##  2 Method

We use two separate methods for generating synthetic magnitudes from blackbody
spectra and for fitting them back to blackbody spectra (each method performs
both operations). The first method makes use of the Astrolib PySynphot package
(hereafter referred to only as pysynphot; STScI Development Team 2013), and
the second uses the Light Curve Fitting package (hereafter,
lightcurve_fitting; Hosseinzadeh & Gomez 2021), Version 0.4.0.

The main difference between the methods is in the way the blackbody spectrum
is fit to the data (see below). However, there are also differences in the
magnitudes generated by each method for given blackbodies (Appendix A and
Figure 8). Given these differences, we use each method to fit only data
simulated with that same method.

Figure 2: Example MCMC fits to simulated magnitudes of a 30,000 K, 1,000 R☉
blackbody (black circles) assuming 0.05-magnitude photometric uncertainties,
using only optical (in red) vs. optical+ultraviolet (in blue) bands (100
random models drawn from the posterior distribution are shown). The insets
show the corner plots for each MCMC fit, with contour lines denoting 90%
bounds, and dotted lines denoting the true values. The top and bottom plots
are two different realizations of simulated data, each fit also with a
different method (see text for details). Including ultraviolet data reduces
both the errors and the uncertainties significantly. Figure 3: Median (of
100-realization ensembles) best-fit measured temperature vs. the true
(simulated) temperature for different methods, priors and band combinations as
noted, using photometric uncertainties of 0.05 magnitudes. The fits using a
log-uniform prior (solid lines) produce more precise median temperatures
compared to those using a uniform prior (dashed lines), when using optical
data alone (left panel). When adding space-based ultraviolet data (right
panel), both priors perform equally well. Figure 4: Top panels: Best-fit
measured temperature vs. the true (simulated) temperature for different band
combinations as noted, using simulated measurement uncertainties of 0.05
magnitudes and a log-uniform prior. The solid lines denote the ensemble median
best-fit temperature, and the shaded regions denote the 16th–84th ensemble
percentile. Results for both methods are shown. Bottom panels: Same as top,
but for the difference between the measured and true temperatures. The mean
ensemble upper and lower fit uncertainties are shown in dotted lines. Figure
5: Same as Figure 4 but using simulated photometric uncertainties of 0.1
magnitudes.

###  2.1 Generating Synthetic Data

We create spectra of blackbodies with a radius of 1,000 R☉ and temperatures of
5,000 K–80,000 K in 5,000 K increments. For each blackbody, we generate
synthetic spectral energy distribution (SED) magnitudes in the Sloan Digital
Sky Survey (SDSS; Doi et al., 2010) $ugri$ bands, the Johnson-Cousins (Johnson
& Morgan, 1953; Cousins, 1976) $BVRI$ bands, and The Neil Gehrels Swift
Observatory (hereafter Swift; Gehrels et al. 2004) Ultraviolet/Optical
Telescope (Roming et al., 2005) $uvw1$, $uvm2$ and $uvw2$ bands (hereafter
$w1$, $m2$ and $w2$ respectively)111Magnitudes in the $ugri$-bands are
generated in the AB system, while magnitudes in the rest of the bands are
generated in the Vega system..

We generate the data in six band combinations: $gri$ (optical observations),
$ugri$ (optical with ground-based ultraviolet observations), $ugri$ \+ $w1$,
$m2$ and $w2$ (optical with ground- and space-based ultraviolet observations),
and similarly for the Johnson-Cousins bands, $BVRI$, $UBVRI$, and $UBVRI$ \+
$w1$, $m2$ and $w2$. We do not consider infrared bands. Although hot
blackbodies differ in their infrared emission (Fig. 1), in practice their
infrared emission is faint compared to their optical emission, and therefore
more difficult to measure.

In total we produce 32 sets of blackbody simulation parameters (16
temperatures using two methods for generating the magnitudes). For each of
these 32 sets of parameters, we generate two ensembles of 100 SED realizations
each by adding randomly-generated Gaussian noise to each synthetic magnitude,
once with a standard deviation of 0.05 magnitudes and once with a standard
deviation of 0.1 magnitudes222We use fixed standard deviations for all bands
at all temperatures, though in reality, measurement uncertainties could depend
on band and temperature..

###  2.2 Fitting the Synthetic Data to Blackbodies

We use the “forward modelling” technique preferred by Brown et al. (2016),
whereby the model spectrum (in our case, a blackbody) is convolved with the
various filter response curves and then compared to the simulated data with
chi-squared used as the likelihood function and all bands weighted equally. In
the pysynphot method, the fit is done in magnitude vs. wavelength space, while
in the lightcurve_fitting package it is done in flux density ($F_{\nu}$) vs.
frequency space.

In both cases the fit is performed using the Markov Chain Monte Carlo (MCMC)
technique as implemented by the emcee package (Foreman-Mackey et al., 2013).
We use 16 walkers and 600 steps (of which 200 are burn-in steps) to fit both
the temperature and radius of the blackbody to the data. We repeat the fits
with two types of priors: uniform, and log-uniform for both the temperature
(in the range $10^{3}$–$10^{5}$ K), and the radius (in the range $10$–$10^{6}$
R☉). For the log-uniform case, we also test the effect of under-estimating the
photometric uncertainties by reporting half of the true uncertainty to the
fitters. The code used to simulate and fit the blackbodies is publicly
available on Github333https://github.com/arcavi/bb_sims.

In total we fit each of the $32\times 200$ realizations of simulated data in
18 different ways (six band combinations, using two types of priors and under-
estimated uncertainties for one of the priors). Thus we have 115,200 different
fits, which we proceed to analyze.

###  2.3 Nomenclature

Hereafter, we will use the term “error” to denote the difference between a
measured value and a true value, while the term “uncertainty” will denote the
estimated spread of a measured value due to measurement errors or fit
posteriors. For example, if for a 35,000 K blackbody we measure a temperature
of 29,000$\pm$7,000 K, then the error is 6,000 K while the uncertainty is
7,000 K.

##  3 Results

###  3.1 Example Fits

Example fits using the log-uniform prior for a 30,000 K blackbody are shown in
Figure 2. As expected, fitting such a blackbody with optical data alone
results in both inaccurate (i.e. having a large error) and imprecise (i.e.
having a large uncertainty) measurements of the blackbody radius and
temperature, compared to fitting it with optical and ultraviolet data
together. In the top realization in Figure 2, fit with the pysynphot method,
the most likely values are close to the true ones, but the uncertainty in the
optical-only fits are much larger compared to the fits which use the optical
and ultraviolet data together. In the bottom realization, fit with the
lightcurve_fitting method, the optical-only fits show not only larger
uncertainties compared to the combined optical-ultraviolet fits, but are also
farther from the true values (by $\sim$10,000 K).

The difference in accuracy in this example is not due to the different methods
(see below), but rather the different realizations. This is why we generate
ensembles of 100 realizations for each temperature, band combination,
magnitude uncertainty, and method, and look at the statistical properties of
the resulting fits.

###  3.2 Full Set of Fits

For each individual MCMC fit we take the 50th percentile (i.e. median) of the
posterior as the best-fit estimated value. Repeating this for each of the 100
realizations of a given ensemble, produces an ensemble-distribution of best-
fit values for that set of blackbody temperature, band combination, magnitude
uncertainty, and method. Similarly, we take the 16th and 84th percentiles of
each MCMC fit posterior as the lower and upper uncertainty estimates of the
fit respectively444We use the median and 68% credible interval range since it
is equivalent to using the mean $\pm 1\sigma$ range for normally-distributed
results, but does not require assuming the distribution is normal (or even
symmetric)., and also produce an ensemble-distribution of these values per
parameter set.

####  3.2.1 The Effect of Priors

We plot the median of the best-fit temperature ensemble-distributions for each
set of blackbody simulation parameters and each type of prior for the 0.05
magnitude uncertainty simulations and the SDSS SEDs in Figure 3. When using
optical and ultraviolet data (right panel in Figure 3), both priors produce
similar results. However, when using optical data alone (left panel in Figure
3), the log-uniform prior produces more accurate ensemble-median best-fit
temperatures, compared to the uniform prior. The results for the SDSS 0.1
magnitude uncertainties and for both uncertainty values with the Johnson-
Cousins SEDs are similar and are presented in Figures 9, 10 and 11. We
conclude that the log-uniform prior produces more precise results when fitting
optical data, and we proceed to analyze only those fits in the remainder of
this work.

####  3.2.2 Full Results for the Log-Uniform Prior

We plot the median and 16th–84th percentile of the ensemble-distributions for
each set of blackbody simulation parameters in Figures 4 and 5 for the SDSS
SEDs (the results for the Johnson-Cousins SEDs are similar and are presented
in Figures 12 and 13).

So, for example, if the true temperature of the object being measured is
40,000 K, then $gri$-band measurements with an uncertainty of 0.05 magnitudes,
yield best-fit temperatures in the range $\sim$30,000–60,000 K (68% bounds;
top left panel of Figure 4). The fit uncertainties in this case are
$\sim$15,000–20,000 K (bottom left panel of Figure 4). Incorporating ground-
and space-based ultraviolet measurements reduces the range of best fit
temperatures to just a few hundred K around the true temperature (top right
panel of Figure 4). The fit uncertainties are also (correctly) reduced to a
few hundred K in this case (bottom right panel of Figure 4).

####  3.2.3 The Effect of Underestimating the Photometric Uncertainties

We plot the simulation results in the same way as above for the fits where the
reported uncertainty is half of the true one in Figures 14 – 17. We find that
there is no significant difference compared to the case where the
uncertainties are estimated correctly. The only apparent difference is that
underestimating the photometric uncertainties causes the pysynphot method to
underestimates the derived temperature uncertainties.

##  4 Analysis

Having established that the log-uniform prior performs better compared to the
uniform prior and that under-estimating the photometric uncertainties does not
have a strong effect on the results, our analysis hereafter focuses only on
the fits using the log-uniform prior and having the correct uncertainty
estimations.

###  4.1 Errors in Temperature Estimation

We find that for 0.05 magnitude uncertainties (Fig. 4), relying on $gri$-band
data alone can result in several thousand K errors in temperature (in both
directions) already at 20,000 K. At around 40,000 K these errors increase to
-10,000 K and +20,000 K. At temperatures of 80,000 K, systematic
underestimates of up to 20,000 K can occur. For 0.1 magnitude uncertainties
(Fig. 5), at temperatures of 30,000 K, overestimates of 20,000 K can occur,
while at 80,000 K, underestimates reach 40,000 K. Adding $u$-band observations
slightly decreases these errors, but not significantly.

In all of these cases, the reported fit uncertainties (dotted lines in the
bottom panels of Figures 4 and 5) roughly encompass the error correctly.
However, for blackbodies hotter than about 50,000 K, the fit uncertainties
remain symmetric while the error is systematically towards underestimating the
temperature (as expected when sampling the blackbody deep in its
Rayleigh–Jeans tail). Similar results are seen for the Johnson-Cousins SEDs
(Figures 12 and 13).

It is only when adding data from the Swift bands that the errors decrease
significantly, and the fit uncertainties encompass the errors correctly at all
temperatures.

In all cases studied there are almost no differences between the two fitting
methods.

###  4.2 The True-Temperature Posterior For Optical Fits

Unfortunately, ultraviolet observations are expensive and not always possible.
Therefore it desirable to know the correct uncertainty when measuring
temperatures with optical data alone. This is essentially the action of
inverting the top left plots of Figures 4, 5, 12 and 13.

In other words, assume we are only able to obtain $gri$-band data, we fit a
blackbody to them, and we obtain a best-fit temperature of 30,000 K - what
true temperatures could have led to that measurement, given a known magnitude
uncertainty?

The answer to that question, for $gri$-band data with 0.05 magnitude
uncertainties fit to a blackbody using the lightcurve_fitting method, is
presented in Figure 6. So, for example, if we obtained a best-fit temperature
in the range 30,000–35,000 K, the true temperature is indeed likely in that
range, but could be as low as 20,000 K and there is a probability tail going
up to and above 60,000 K (6th histogram from the bottom in Figure 6). The
results are similar for $BVRI$-band data and when using the pysynphot method.
These are shown, together with 0.1 magnitude uncertainty data, in Figure 18.

Figure 6: Distributions of the true temperatures (up to the limit of what was
simulated - 80,000 K) that lead to various measured temperatures (in 5,000 K
bins) when using $gri$-band data with 0.05 magnitude photometric uncertainties
fit to a blackbody with the lightcurve_fitting method. Above measured
temperatures of 25,000 K, the real temperature uncertainty becomes large.

The 16th, 50th and 84th percentiles of the true temperature distributions that
produce a given measured temperature range are presented (as the median and
lower and upper bounds) in Tables 1 (for the lightcurve_fitting method) and 2
(for the pysynphot method). The data behind these distributions are publicly
available on GitHub555https://github.com/arcavi/bb_sims. Continuing our
example where we obtained a best-fit temperature in the range 30,000–35,000 K
by fitting $gri$-band data with 0.05 magnitude uncertainties, the true
temperature could be between 25,000 K and 55,000 K (68% confidence bounds)
with a median of 35,000 K.

This is a large uncertainty, which needs to be taken into account when fitting
hot blackbodies using optical data alone. It becomes especially important for
optical-only measured temperatures $\gtrsim$ 25,000 K.

Table 1: Median and 68% confidence bounds of the true temperatures that
produce various measured temperatures in the lightcurve_fitting method for
different band and magnitude uncertainty combinations.

Measured Temperature | True Temperature
---|---
| For a $gri$-Band Measurement | For a $BVRI$-Band Measurement
| 0.05 Mag Uncertainty | 0.1 Mag Uncertainty | 0.05 Mag Uncertainty | 0.1 Mag Uncertainty
[$10^{3}$ K] | [$10^{3}$ K] | [$10^{3}$ K] | [$10^{3}$ K] | [$10^{3}$ K]
5–10 | $5.0_{-0.0}^{+5.0}$ | $5.0_{-0.0}^{+5.0}$ | $10.0_{-5.0}^{+0.0}$ | $5.0_{-0.0}^{+5.0}$
10–15 | $10.0_{-0.0}^{+5.0}$ | $15.0_{-5.0}^{+0.0}$ | $15.0_{-5.0}^{+0.0}$ | $12.5_{-2.5}^{+2.5}$
15–20 | $15.0_{-0.0}^{+5.0}$ | $20.0_{-5.0}^{+15.0}$ | $20.0_{-5.0}^{+0.0}$ | $20.0_{-5.0}^{+5.0}$
20–25 | $25.0_{-5.0}^{+5.0}$ | $35.0_{-15.0}^{+16.0}$ | $25.0_{-5.0}^{+0.0}$ | $25.0_{-5.0}^{+15.0}$
25–30 | $30.0_{-5.0}^{+10.0}$ | $35.0_{-10.0}^{+29.2}$ | $30.0_{-5.0}^{+10.0}$ | $35.0_{-10.0}^{+20.0}$
30–35 | $35.0_{-10.0}^{+20.0}$ | $40.0_{-15.0}^{+25.6}$ | $35.0_{-10.0}^{+10.0}$ | $45.0_{-18.2}^{+23.2}$
35–40 | $45.0_{-10.0}^{+20.0}$ | $50.0_{-20.0}^{+20.0}$ | $40.0_{-10.0}^{+15.0}$ | $45.0_{-10.0}^{+20.0}$
40–45 | $50.0_{-15.0}^{+20.0}$ | $47.5_{-20.3}^{+22.5}$ | $45.0_{-10.0}^{+25.0}$ | $50.0_{-15.4}^{+20.0}$
45–50 | $50.0_{-10.0}^{+25.0}$ | $55.0_{-20.0}^{+20.0}$ | $52.5_{-12.5}^{+17.5}$ | $50.0_{-15.0}^{+20.0}$
50–55 | $55.0_{-15.0}^{+15.2}$ | $60.0_{-25.0}^{+15.0}$ | $50.0_{-10.0}^{+20.0}$ | $55.0_{-20.0}^{+15.0}$
55–60 | $60.0_{-15.0}^{+15.0}$ | $55.0_{-15.0}^{+20.0}$ | $57.5_{-12.9}^{+17.5}$ | $60.0_{-20.0}^{+15.0}$
60–65 | $60.0_{-13.2}^{+15.0}$ | $60.0_{-20.0}^{+15.0}$ | $60.0_{-11.0}^{+10.0}$ | $60.0_{-18.0}^{+10.0}$
65–70 | $60.0_{-15.0}^{+15.0}$ | $60.0_{-15.0}^{+15.0}$ | $65.0_{-15.0}^{+10.0}$ | $65.0_{-15.8}^{+15.0}$
70–75 | $65.0_{-11.4}^{+10.0}$ | $65.0_{-20.0}^{+10.0}$ | $65.0_{-10.8}^{+10.8}$ | $65.0_{-10.0}^{+10.0}$
75–80 | $70.0_{-20.0}^{+10.0}$ | $60.0_{-10.2}^{+3.4}$ | $70.0_{-15.0}^{+10.0}$ | $70.0_{-14.0}^{+10.0}$

Note. — Simulations only go up to 80,000 K. The true upper bound is likely
much higher for measured temperatures above 40,000 K (the true temperature
could actually be unbounded from above in certain regimes, see text for
details).

Our results are truncated at 80,000 K because those are the highest
temperatures we simulated, but it is evident from the flattening of the curve
in the top left panels of Figures 4, 5, 12 and 13, that any optical
measurement yielding a best fit temperature of 60,000 K or above for 0.05
magnitude uncertainties, and 40,000 K or above for 0.1 magnitude
uncertainties, actually sets only a lower limit on the true temperature.

###  4.3 Errors in Bolometric Luminosity Estimation

In many cases, the blackbody temperature $T$ and radius $R$ are used to
estimate the bolometric luminosity through
$L_{bol}=4{\pi}R^{2}\sigma_{SB}T^{4}$ (with $\sigma_{SB}$ the Stephan-
Boltzmann constant). One would expect that large errors in temperature would
translate to order-of-magnitude errors in bolometric luminosity. However, as
seen in the example fits in Figure 2, there is typically an anti-correlated
degeneracy between temperature and radius which cancels some of the error in
each parameter when estimating the bolometric luminosity.

Figure 7 shows the ratio between the measured bolometric luminosity and the
true bolometric luminosity for various SDSS and ultraviolet band combinations.
When relying on optical data alone, the luminosity of blackbodies up to
$\sim$60,000 K is overestimated on average. For 0.05 magnitude uncertainties,
the luminosity overestimation reaches a factor of $\sim$3 (84th ensemble
percentile). At higher temperatures, beyond $\sim$60,000 K, the preference
shifts to underestimating the luminosity by factors of $\sim$3 (16th
percentile). For 0.1 magnitude uncertainties, the luminosity over- and
underestimates reach factors of $\sim$4–5 (at the 84th and 16th percentiles
respectively). Adding space-based ultraviolet bands reduces these errors
significantly with the ensemble-average luminosities tracking the true
luminosity at all temperatures tested. Similar results are seen for the
Johnson-Cousins SEDs (Fig. 19). Like in the temperature estimation, there are
almost no differences between the two methods (except at higher temperatures
in the $gri$-band fits where the lightcurve_fitting method has a longer tail
towards underestimation of the true luminosity).

Figure 7: Ratios between the measured and true bolometric luminosities at
various temperatures for different SEDs as noted, using photometric
uncertainties of 0.05 magnitudes (left) and 0.1 magnitudes (right). The solid
lines use the bolometric luminosity calculated from the ensemble median best-
fit temperature and radius, and the shaded regions denote the ensemble
16th–84th percentile range.

##  5 Summary and Conclusions

When limited to optical data, large errors in blackbody temperatures can
occur, reaching $\sim$10,000 K at 30,000–40,000 K (depending on the
measurement uncertainties). Beyond 40,000–60,000 K it is only possible to
obtain lower limits on the true temperature. It is important to consider these
errors whenever an optical measurement yields a temperature $\gtrsim$ 25,000
K. We calculate realistic uncertainties for such measurements and present them
in Tables 1 and 2. These results refer to fits using log-uniform priors.
Uniform priors produce even less precise results, and are not discussed
further. Underestimating the photometric uncertainties by a factor of two has
almost no effect on the results (especially when using the lightcurve_fitting
package).

Despite the large errors in temperature measurements of hot blackbodies with
optical data alone, the error on the bolometric luminosity calculated from
such fits is not orders of magnitude as one might naively expect from the
$L_{bol}{\propto}T^{4}$ dependence. Temperature over-estimations partially
cancel with radius underestimations (and vice versa), producing bolometric-
luminosity errors up to factors of 3–5 in 68% of cases.

Adding ground-based ultraviolet observations reduces these errors somewhat,
but space-based ultraviolet observations are required to reduce these errors
significantly to less than a few hundred K in temperature and 10%-20% in
bolometric luminosity.

There are no major differences between the results obtained when using the
pysynphot method compared to the lightcurve_fitting method. However, the
lightcurve_fitting method is significantly faster.

Our results relate to pure and single blackbodies. Departures from such
spectra due to absorption and emission features, extinction, or other types of
underlying spectra altogether, will induce even larger errors. In addition,
errors in distance estimation will cause errors in determining the blackbody
radius, which in turn will cause errors in determining the bolometric
luminosity. Given the degeneracy between temperature and radius fitting,
distance errors could cause additional errors also in temperature
determination.

Observations in the ultraviolet, currently possible in large numbers only with
the Swift observatory, and soon also with the Ultraviolet Transient Astronomy
Satellite (ULTRASAT; Sagiv et al., 2014), are crucial for constraining the
emission properties, and hence physics, of high-temperature astrophysical
objects and events. When ultraviolet observations are not available, it is
important to be aware of the uncertainties quantified here.

We thank G. Hosseinzadeh for assistance in using the lightcurve_fitting
package and for helpful comments, and D. A. Howell for helpful comments. I.A.
is a CIFAR Azrieli Global Scholar in the Gravity and the Extreme Universe
Program and acknowledges support from that program, from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
program (grant agreement number 852097), from the Israel Science Foundation
(grant number 2752/19), from the United States – Israel Binational Science
Foundation (BSF), and from the Israeli Council for Higher Education Alon
Fellowship.

## References

  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., & et al. 2017, ApJ, 848, L12, doi: 10.3847/2041-8213/aa91c9
  * Arcavi (2018) Arcavi, I. 2018, ApJ, 855, L23, doi: 10.3847/2041-8213/aab267
  * Arcavi et al. (2017) Arcavi, I., Hosseinzadeh, G., Brown, P. J., et al. 2017, ApJ, 837, L2, doi: 10.3847/2041-8213/aa5be1
  * Breeveld et al. (2011) Breeveld, A. A., Landsman, W., Holland, S. T., et al. 2011, in American Institute of Physics Conference Series, Vol. 1358, Gamma Ray Bursts 2010, ed. J. E. McEnery, J. L. Racusin, & N. Gehrels, 373–376, doi: 10.1063/1.3621807
  * Brown et al. (2016) Brown, P. J., Breeveld, A., Roming, P. W. A., & Siegel, M. 2016, AJ, 152, 102, doi: 10.3847/0004-6256/152/4/102
  * Cousins (1976) Cousins, A. W. J. 1976, MmRAS, 81, 25
  * Dai et al. (2018) Dai, L., McKinney, J. C., Roth, N., Ramirez-Ruiz, E., & Miller, M. C. 2018, ApJ, 859, L20, doi: 10.3847/2041-8213/aab429
  * Doi et al. (2010) Doi, M., Tanaka, M., Fukugita, M., et al. 2010, AJ, 139, 1628, doi: 10.1088/0004-6256/139/4/1628
  * Faran et al. (2018) Faran, T., Nakar, E., & Poznanski, D. 2018, MNRAS, 473, 513, doi: 10.1093/mnras/stx2288
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Gehrels et al. (2004) Gehrels, N., Chincarini, G., Giommi, P., et al. 2004, ApJ, 611, 1005, doi: 10.1086/422091
  * Gezari (2021) Gezari, S. 2021, ARA&A, 59, 21, doi: 10.1146/annurev-astro-111720-030029
  * Hosseinzadeh & Gomez (2021) Hosseinzadeh, G., & Gomez, S. 2021, Light Curve Fitting, v0.3.0, Zenodo, doi: 10.5281/zenodo.5520603
  * Johnson & Morgan (1953) Johnson, H. L., & Morgan, W. W. 1953, ApJ, 117, 313, doi: 10.1086/145697
  * Mockler et al. (2019) Mockler, B., Guillochon, J., & Ramirez-Ruiz, E. 2019, ApJ, 872, 151, doi: 10.3847/1538-4357/ab010f
  * Nakar & Sari (2010) Nakar, E., & Sari, R. 2010, ApJ, 725, 904, doi: 10.1088/0004-637X/725/1/904
  * Piran et al. (2015) Piran, T., Svirski, G., Krolik, J., Cheng, R. M., & Shiokawa, H. 2015, ApJ, 806, 164, doi: 10.1088/0004-637X/806/2/164
  * Rabinak & Waxman (2011) Rabinak, I., & Waxman, E. 2011, ApJ, 728, 63, doi: 10.1088/0004-637X/728/1/63
  * Roming et al. (2005) Roming, P. W. A., Kennedy, T. E., Mason, K. O., et al. 2005, Space Sci. Rev., 120, 95, doi: 10.1007/s11214-005-5095-4
  * Rubin & Gal-Yam (2017) Rubin, A., & Gal-Yam, A. 2017, ApJ, 848, 8, doi: 10.3847/1538-4357/aa8465
  * Ryu et al. (2020) Ryu, T., Krolik, J., & Piran, T. 2020, ApJ, 904, 73, doi: 10.3847/1538-4357/abbf4d
  * Sagiv et al. (2014) Sagiv, I., Gal-Yam, A., Ofek, E. O., et al. 2014, AJ, 147, 79, doi: 10.1088/0004-6256/147/4/79
  * Sapir & Waxman (2017) Sapir, N., & Waxman, E. 2017, ApJ, 838, 130, doi: 10.3847/1538-4357/aa64df
  * Shussman et al. (2016) Shussman, T., Waldman, R., & Nakar, E. 2016, arXiv e-prints, arXiv:1610.05323.  https://arxiv.org/abs/1610.05323
  * STScI Development Team (2013) STScI Development Team. 2013, pysynphot: Synthetic photometry software package.  http://ascl.net/1303.023
  * Valenti et al. (2016) Valenti, S., Howell, D. A., Stritzinger, M. D., et al. 2016, MNRAS, 459, 3939, doi: 10.1093/mnras/stw870
  * van Velzen et al. (2020) van Velzen, S., Holoien, T. W. S., Onori, F., Hung, T., & Arcavi, I. 2020, Space Sci. Rev., 216, 124, doi: 10.1007/s11214-020-00753-z

##  Appendix A Differences in Synthetic Magnitudes Generated by the Two
Packages

We generate synthetic magnitudes for blackbodies with a radius of 1,000 R☉ and
temperatures of 5,000, 10,000, 20,000, 30,000 and 60,000 K using both the
pysynphot and lightcurve_fitting packages. Figure 8 shows the differences in
magnitudes generated by the two methods.

Figure 8: Differences in blackbody absolute magnitudes generated by the two
packages used here (pysynphot and lightcurve_fitting) for various bands at
various temperatures.

Some of these difference may be due to different filter response curves used
by each package. However, the largest differences (up to 0.2 magnitudes for
some temperatures) are seen in the $w2$ band even though both packages use the
same Breeveld et al. (2011) response curves for the Swift $w1$, $m2$ and $w2$
bands. Therefore there are likely other factors responsible for these
inconsistencies.

##  Appendix B Results and Analysis for Additional Scenarios

We present in Figures 9, 10 and 11 our prior comparison results for the SDSS
SEDs with 0.1 magnitude errors and the Johnson-Cousins SEDs with both the 0.05
and 0.1 magnitude errors. The results are similar to those presented and
analyzed in the main text for the SDSS SEDs with 0.05 magnitude errors.

Figure 9: Same as Figure 3 but using simulated measurement uncertainties of
0.1 magnitudes. Figure 10: Same as Figure 3 but for Johnson-Cousins bands.
Figure 11: Same as Figure 3 but for Johnson-Cousins bands and using simulated
measurement uncertainties of 0.1 magnitudes.

We present in Figures 12 and 13 our full simulation results for the Johnson-
Cousins SEDs using the log-uniform prior. The results are similar to those
presented and analyzed in the main text for the SDSS SEDs with the same prior.

Figure 12: Same as Figure 4 but for Johnson-Cousins bands. Figure 13: Same as
Figure 4 but for Johnson-Cousins bands and using simulated measurement
uncertainties of 0.1 magnitudes.

In Figures 14 – 17 we present the full simulation results for the fits with
uncertainties underestimated by a factor of two. We find no major differences
compared to the fits with correctly estimated uncertainties, except that the
uncertainties in the fitted temperature are underestimated when using the
pysynphot method.

Figure 14: Same as Figure 4 but for fits with uncertainties underestimated by
a factor of two. Figure 15: Same as Figure 14 but using simulated measurement
uncertainties of 0.1 magnitudes. Figure 16: Same as Figure 14 but for Johnson-
Cousins bands. Figure 17: Same as Figure 14 but for Johnson-Cousins bands and
using simulated measurement uncertainties of 0.1 magnitudes.

The posterior distributions of true temperatures that produce various measured
temperatures for the different optical SEDs, assumed magnitude uncertainties,
and methods (using log-uniform priors) are shown in Figure 18. The median and
$1\sigma$ bounds of these distributions, for the pysynphot method are
presented in Table 2.

Figure 18: Same as Figure 6 but for both methods as shown in the legend, and for both SDSS and Johnson-Cousins optical bands, and various magnitude uncertainties, as shown in the titles. Table 2: Same as Table 1 but for the pysynphot method. Measured Temperature | True Temperature
---|---
| For a $gri$-Band Measurement | For a $BVRI$-Band Measurement
| 0.05 Mag Uncertainty | 0.1 Mag Uncertainty | 0.05 Mag Uncertainty | 0.1 Mag Uncertainty
[$10^{3}$ K] | [$10^{3}$ K] | [$10^{3}$ K] | [$10^{3}$ K] | [$10^{3}$ K]
5–10 | $5.0_{-0.0}^{+5.0}$ | $5.0_{-0.0}^{+5.0}$ | $5.0_{-0.0}^{+5.0}$ | $5.0_{-0.0}^{+5.0}$
10–15 | $10.0_{-0.0}^{+5.0}$ | $10.0_{-0.0}^{+5.0}$ | $10.0_{-0.0}^{+5.0}$ | $10.0_{-0.0}^{+5.0}$
15–20 | $15.0_{-0.0}^{+5.0}$ | $15.0_{-0.0}^{+10.0}$ | $15.0_{-0.0}^{+5.0}$ | $20.0_{-5.0}^{+5.0}$
20–25 | $25.0_{-5.0}^{+5.0}$ | $25.0_{-8.6}^{+10.0}$ | $22.5_{-2.5}^{+2.5}$ | $25.0_{-5.0}^{+10.0}$
25–30 | $30.0_{-5.0}^{+5.0}$ | $30.0_{-10.0}^{+15.0}$ | $30.0_{-5.0}^{+5.0}$ | $25.0_{-5.0}^{+15.0}$
30–35 | $35.0_{-10.0}^{+10.0}$ | $35.0_{-10.0}^{+20.0}$ | $35.0_{-6.0}^{+6.0}$ | $35.0_{-10.0}^{+15.0}$
35–40 | $40.0_{-10.0}^{+15.0}$ | $40.0_{-15.0}^{+25.0}$ | $35.0_{-5.0}^{+15.0}$ | $40.0_{-10.0}^{+20.0}$
40–45 | $45.0_{-10.0}^{+15.0}$ | $45.0_{-15.0}^{+20.0}$ | $45.0_{-10.0}^{+20.0}$ | $50.0_{-20.0}^{+17.0}$
45–50 | $50.0_{-15.0}^{+18.4}$ | $50.0_{-20.0}^{+20.4}$ | $45.0_{-10.0}^{+20.0}$ | $50.0_{-15.0}^{+20.0}$
50–55 | $55.0_{-15.0}^{+15.0}$ | $60.0_{-20.0}^{+15.0}$ | $55.0_{-10.0}^{+14.8}$ | $55.0_{-10.0}^{+16.6}$
55–60 | $60.0_{-15.0}^{+15.0}$ | $60.0_{-20.0}^{+15.0}$ | $55.0_{-10.0}^{+20.0}$ | $55.0_{-15.0}^{+20.0}$
60–65 | $60.0_{-15.0}^{+15.0}$ | $60.0_{-15.0}^{+15.0}$ | $60.0_{-10.0}^{+15.0}$ | $60.0_{-19.8}^{+15.0}$
65–70 | $65.0_{-15.0}^{+10.0}$ | $60.0_{-15.0}^{+15.0}$ | $62.5_{-7.5}^{+12.5}$ | $65.0_{-15.0}^{+10.0}$
70–75 | $67.5_{-17.5}^{+7.5}$ | $70.0_{-10.0}^{+10.0}$ | $65.0_{-15.0}^{+10.0}$ | $65.0_{-10.0}^{+15.0}$
75–80 | $70.0_{-12.0}^{+10.0}$ | $62.5_{-6.9}^{+11.9}$ | $75.0_{-10.0}^{+5.0}$ | $75.0_{-17.8}^{+5.0}$

The bolometric luminosity errors for the Johnson-Cousins SEDs (assuming log-
uniform priors) are presented in Figure 19. These results are similar to the
results presented in the main text for the SDSS SEDs using the same priors.

Figure 19: Same as Figure 7 but but for Johnson-Cousins bands using 0.05
magnitude (left) and 0.1 magnitude (right) simulated measurement
uncertainties.