# CQE: A Comprehensive Quantity Extractor

Satya Almasian, Vivian Kazakova∗, Philip Göldner  These authors contributed
equally to this work. Michael Gertz
Institute of Computer Science, Heidelberg University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Quantities are essential in documents to describe factual information. They
are ubiquitous in application domains such as finance, business, medicine, and
science in general. Compared to other information extraction approaches,
interestingly only a few works exist that describe methods for a proper
extraction and representation of quantities in text.
In this paper, we present such a comprehensive quantity extraction framework
from text data. It efficiently detects combinations of _values_ and _units_ ,
the behavior of a quantity (e.g., rising or falling), and the _concept_ a
quantity is associated with. Our framework makes use of dependency parsing and
a dictionary of units, and it provides for a proper normalization and
standardization of detected quantities. Using a novel dataset for evaluation,
we show that our open source framework outperforms other systems and – to the
best of our knowledge – is the first to detect concepts associated with
identified quantities. The code and data underlying our framework are
available at https://github.com/vivkaz/CQE.

##  1 Introduction

Quantities are the main tool for conveying factual and accurate information.
News articles are filled with social and financial trends, and technical
documents use measurable values to report their findings. Despite their
significance, a comprehensive system for quantity extraction and an evaluation
framework to compare the performance of such systems is not yet at hand. In
the literature, a few works directly study quantity extraction, but their
focus is limited to physical and science domains Foppiano et al. (2019).
Quantity extraction is often part of a larger system, where identification of
quantities is required to improve numerical understanding in retrieval or
textual entailment tasks Roy et al. (2015); Li et al. (2021); Sarawagi and
Chakrabarti (2014); Banerjee et al. (2009); Maiya et al. (2015). Consequently,
their performance is measured based on the downstream task, and the quality of
the extractor, despite its contribution to the final result, is not separately
evaluated. Therefore, when in need of a quantity extractor, one has to resort
to a number of open source packages, without a benchmark or a performance
guarantee. Since quantity extraction is rarely the main objective, the
capabilities of the available systems and their definition of quantity vary
based on the downstream task. As a result, the context information about a
quantity is reduced to the essentials of each system. Most systems consider a
quantity as a number with a measurable and metric _unit_ Foppiano et al.
(2019). However, outside of scientific domains any noun phrase describing a
_value_ is a potential _unit_ , e.g., “5 bananas”. Moreover, a more meaningful
representation of quantities should include their behaviour and associated
_concepts_. For example, in the sentence “DAX fell 2% and S&P gained more than
2%”, the _value/unit_ pair $\langle$_2, percentage_ $\rangle$ indicates two
different quantities in association with different concepts, DAX and S&P, with
opposite behaviours, _decreasing_ and _increasing_ , subtleties not captured
by simplified models. In this paper, we present a comprehensive quantity
extraction (CQE) framework. Our system is capable of extracting standardized
_values_ , physical and non-physical _units_ , _changes_ or trends in the
values and _concepts_ associated with detected values. Furthermore, we
introduce NewsQuant, a new benchmark dataset for quantity extraction,
carefully selected from a diverse set of news articles in the categories of
economics, sports, technology, cars, science, and companies. Our system
outperforms other libraries and extends on their capabilities to extract
_concepts_ associated with values. Our software and data are publicly
available. By introducing a strong baseline and a novel dataset, we aim to
motivate further research in this field.

##  2 Related Work

In literature, quantity extraction is mainly a component of a larger system
for textual entailment or search. The only work that solely focuses on
quantity extraction is Grobid-quantities Foppiano et al. (2019), which uses
three Conditional Random Field models in a cascade to find _value/unit_ pairs
and to determine their relation, where the _units_ are limited to the
scientific domain, a.k.a.  _SI units_.

Roy et al.’s Roy et al. (2015) definition of a quantity is closer to ours and
is based on Forbus’ theory Forbus (1984). A quantity is a (_value_ , _unit_ ,
_change_) triplet, and noun-based units are also considered. Extraction is
performed as a step in their pipeline for quantity reasoning in terms of
textual entailment. Although they only evaluate on textual entailment, the
extractor is released as part of the CogComp natural language processing
libraries, under the name Illinois
Quantifier111https://github.com/CogComp/cogcomp-nlp/tree/master/quantifier
Last accessed: April 15, 2023.

Two prominent open source libraries for quantity extraction are (a)
Recognizers-Text Huang et al. (2017); Chen et al. (2023) from Microsoft and
(b) Quantulum3 222https://github.com/nielstron/quantulum3 Last accessed: April
15, 2023. Recognizers-Text uses regular expressions for the resolution of
numerical and temporal entities in 10 languages. The system has separate
models for the extraction of _value/unit_ pairs for percentages, age,
currencies, dimensions, and temperatures and is limited to only these quantity
types. Moreover, it cannot proactively distinguish the type of quantity for
extraction and the user has to manually select the correct model. Quantulum3
uses regular expression to extract quantities and a dictionary of _units_ for
normalization. For _units_ with similar surface forms, a classifier based on
Glove embeddings Pennington et al. (2014) is used for disambiguation, e.g.,
“pound” as weight or currency. Recognizers-Text is used in the work of Li et
al. (2021) to demonstrate quantity search, where the results are visualized in
the form of tables or charts. They define quantity facts as triplets of
(_related, value & unit, time_). _Related_ is the quantitative related
information, close to our definition of _concept_. However, it is not part of
their quantity model but rather extracted separately using rules. They utilize
the quantity facts for visualization of results but do not evaluate their
system or the quantity extraction module. QFinder Almasian et al. (2022) uses
Quantulum3 in a similar way to demonstrate quantity search on news articles,
but does not comment on the extractor’s performance.

A number of other works utilize quantity extraction as part of their system.
MQSearch Maiya et al. (2015) extracts quantities with a set of regular
expressions for a search engine on numerical information. Qsearch Ho et al.
(2019) is another quantity search system, based on quantity facts extracted
with the Illinois Quantifier. The works by Banerjee et al. (2009); Sarawagi
and Chakrabarti (2014) focus on scoring quantity intervals in census data and
tables.

##  3 Extraction of Quantities

In the following, we describe our quantity representation model and detail our
extraction technique.

###  3.1 Quantity Representation

In general, anything that has a count or is measurable is considered a
quantity. We extend upon the definition by Roy et al. (2015) to include
concepts and represent a quantity by a tuple $\langle v,u,ch,cn\rangle$ with
the following components:

  1. 1.

_Value ( $v$):_ A real number or a range of values, describing a magnitude,
multitude, or duration, e.g., “the car accelerates from 0 to 72 km/h”, has a
range of $v=(0,72)$ and, “the car accelerated to 72 km/h” has a single value
$v=72$. _Values_ come in different magnitudes, often denoted by prefixes, and
sometimes containing fractions, e.g., “He earns 10k euros” $\to$ $v=10000$, or
“1/5 th of his earnings”$\to$ $v=0.2$.

  2. 2.

_Unit ( $u$):_ A noun phrase defining the atomic unit of measure. _Units_ are
either part of a predefined set of known scientific and monetary types, or in
a more general case, are noun phrases that refer to the multitude of an
object, e.g., “2 apples” $\to$ $u=apple$. The predefined set corresponds
either to (a) _scientific units_ for measurement of physical attributes (e.g.,
“2km” has the _scientific unit_ ($u=kilometre$)), or (b) _currencies_ , as the
unit of money (e.g., “10k euros” refers to a currency). Predefined _units_ can
have many textual or symbolic surface forms, e.g., “euro”, “EUR”, or “€”, and
their normalization is a daunting task. Sometimes the surface forms coincide
with other units, resulting in ambiguity that can only be resolved by knowing
the context, e.g., “She weighs 50 pounds”, is a measure of weight ($u=$pound-
mass) and not a currency.

  3. 3.

_Change ( $ch$):_ The modifier of the quantity _value_ , describing how the
_value_ is changing, e.g., “roughly 35$” is describing an approximation. Roy
et al. (2015) introduce four categories for _change_ : $=$ (equal), $\sim$
(approximate), $>$ (more than), and $<$ (less than). These categories are
mainly describing the bounds for a quantity. We extend this definition by
accounting for trends and add two more categories: $up$ and $down$ for
increasing and decreasing trends, e.g., “DAX fell 2%” indicates a downward
trend ($ch=down$), while “He weighs more than 50kg” is indicating a bound
($ch=\text{`>`}$).

  4. 4.

_Concept ( $cn$):_ _Concepts_ are either properties being measured or entities
that the _value_ is referring to or is acting upon. In the phrase “DAX fell
2%” the quantity is measuring the worth of $cn=DAX$ or in “The BMW Group is
investing a total of $200 million ” the investment is being made by $cn=BMW\
Group$. Sometimes a _concept_ is distributed in different parts of a sentence,
e.g., “The iPhone 11 has 64GB of storage. ” $\to cn=iPhone\ 11,storage$. A
_concept_ may or may not be present, e.g., “200 people were at the concert”
has no concept.

###  3.2 Quantity Extraction

Similar to previous work, we observed that quantities often follow a recurring
pattern. But instead of relying on regular expressions, we take advantage of
linguistic properties and dependency parsing. The input of our system is a
sentence, and the output is a list of detected quantities.

Example 1: “In Europe, German DAX fell 0.4 pc, while the CAC40 in France
gained 0.1.” results in

  * •

$\langle v=0.4,\ u=percentage,$
$ch=down,\ cn=(German,DAX)\rangle$

  * •

$\langle v=0.1,\ u=percentage,$
$ch=up,\ cn=(CAC40,France)\rangle$.

The extraction is performed in five stages, as described next.

####  3.2.1 Pre-processing

The pre-processing stage includes the removal of unnecessary punctuations,
e.g., “m.p.h” $\to$ “mph”, the addition of helper tokens, and other text
cleaning steps. Numerals that do not fit the definition of a quantity, such as
phone numbers and dates, are detected with regular expressions disregarded in
further steps.

####  3.2.2 Tokenization

We perform a custom task-specific word tokenization. Our tokenizer is aware of
separator patterns in _values_ and _units_ and avoids between-word splitting.
For example, in the sentence “A beetle goes from 0 to 80 km/h in 8 seconds.”,
a normal tokenizer would split _km/h_ $\to$ _(km, /, h)_ but we will keep the
_unit_ token intact. Another example is a numerical token containing
punctuations, e.g., $2.33E\text{-}3$, where naive tokenization changes the
value.

####  3.2.3 Value, Unit, and Change Detection

The tokenized text is matched against a set of rules based on a dependency
parsing tree and POS tags. A set of 61 rules were created based on patterns
observed and by studying previous work. The comprehensive list of all rules is
found in the repository of our project. The rules are designed to find tokens
associated with _value_ , _unit_ , and _change_. _Value/unit_ pairs are often
sets of numbers and nouns, numbers and symbols, or number and adjectives in
various sentence structures. For ranges, the rules become more complex, as
lower and upper bounds need to be identified using relational keywords such as
“from… to” or “between”. _Changes_ are often adjectives or verbs that have a
direct relation to a number and modify its value. Sometimes symbols before a
number are also an indication of a _change_ , e.g., “$\sim 10$” describes an
approximation. In general, there are six _change_ categories. $\sim$ for
approximate equality, $=$ for exact equality, $>$ for greater than bounds, $<$
for less than bounds, $up$ denoting an increasing or upward trend, and $down$
for decreasing or downward trend. As an example of the extraction, we look at
_value_ , _unit_ and _change_ detection for the two quantities in Example 1.
Note that in this stage the surface forms are detected and not normalized
values, e.g., “pc” versus “percentage”.

The NOUN$\textunderscore$NUM rule detects the surface form for the first
_value/unit_ pair, (0.4, pc). Here, the _value_ has NUM as a POS-tag and is
the immediate syntactic dependent of the _unit_ token, which is a noun or
proper noun.

The LONELY$\textunderscore$NUM rule detects the _value/unit_ pair for the
second quantity, namely (_0.1, None_). If all other rules fail to find a
_value/unit_ pair, this rule detects the number with the POS-tag NUM.

QUANTMOD$\textunderscore$DIRECT$\textunderscore$NUM detects the _change_ , by
looking at the verb or adjective directly before NUM tokens. Here, “fell” is a
trigger word for a downward trend. For example 1, we thus have two extracted
triplets with _value_ , _unit_ , and _change_.

  * •

$\langle v=0.4,\ u=pc,\ ch=fell\rangle$

  * •

$\langle v=0.1,\ u=None,\ ch=gained\rangle$,

In Appendix A.1, more examples are given.
If no _unit_ is detected for a quantity, its context is checked for the
possibility of _shared units_. For the quantity $\langle v=0.1,\ u=None,\
ch=gained\rangle$ in Example 1 ,“percentage” is the derived _unit_ , although
not mentioned in the text. _Shared units_ often occur in similarly structured
sub-clauses or after connector words such as “and”, “while”, or “whereas”. The
similarity between two sub-clauses is computed using the _Levenshtein ratio_
between the structure of clauses. The structure is represented by POS-tags,
e.g., “German DAX fell 0.4 pc” $\to$ “JJ NNP VBD CD NN” and “the CAC40 in
France gained 0.1” $\to$“DT NNP IN NNP VBD CD”. This ratio is between 0 and
100, where larger values indicate higher similarity. If connector words are
present and ratio is larger than 60, the unitless quantity is assigned the
_unit_ of the other sub-clause. e.g., $None$ becomes $pc$.
Finally, the candidate _values_ are filtered by logical rules to avoid false
detection of _non-quantities_ , e.g., in “S&P 500”, 500 is not a quantity.

####  3.2.4 Concept Detection

_Concepts_ are detected in one of the following ways, ordered by priority:

  1. 1.

Keywords, such as _for_ , _of_ , _at_ or _by_ before or after a _value_ point
to a potential concept. For example, “with carbon levels _at_ 1200 parts per
million” results in $cn=(carbon,\ levels)$. The noun and pronouns before and
after such keywords are potential concepts.

  2. 2.

The entire subtree of dependencies with a number as one of the leaf nodes is
inspected to find the closest verb related to the number. If no verb is found,
then the verb connected to the ROOT is selected. The nominal subject of the
verb is considered as the _concept_. In Example 1, both “German DAX” and
“CAC40 in France” are the nominal subjects of the closest verbs to the number
tokens in the text.

  3. 3.

Sometimes values occur in a relative clause that modifies the nominal, e.g.,
“maximum investment per person, which is 50000” $\to$ $cn=(maximum,\
investment,\ per,\ person)$. In such a case, the noun phrase before the
relative clause is the _concept_ , since the relative clause is describing it.

  4. 4.

If the numerical _value_ in a sentence is not associated with the nominal of
the sentence, then it is mostly likely related to the object. Therefore, the
direct object of the verb is also a candidate, e.g., “She gave me a raise of
$1k”, where “raise” is the direct object of the verb.

  5. 5.

Finally, if the _concept_ is not found in the previous step, and there is a
single noun in the sentence, the noun is tagged as the _concept_ , e.g., “a
beetle that can go from 0 to 80 km/h in about 8 seconds, ” $\to$
$cn=(beetle)$.

From the list of candidate tokens for _concepts_ , tokens previously
associated with _units_ and _values_ are filtered and stopwords are remove,
e.g., “CAC40 in France” results$cn=(CAC40,\ France)$. Generally, a _concept_
is represented as a list of tokens.

####  3.2.5 Normalization and Standardization

The final stage is the normalization of _units_ and _changes_ using
dictionaries and standardization of _values_. The _units dictionary_ is a set
of 531 _units_ , their surface forms and symbols gathered from the Quantulum3
library, a dictionary provided by Unified Code for Units of Measure (UCUM)
Lefrançois and Zimmermann (2018), and a list of _units_ from Wikipedia
333https://en.wikipedia.org/wiki/Template:Convert/list_of_units Last accessed:
April 17, 2023. An example of an entry in this dictionary for “euro” is:

    {"euro":
    "surfaces": ["Euro","Euros","euro",
    "euros"],
    "symbols": ["EUR","eur",]}}.

The detected token span of a _unit_ is normalized by matching against the
different surface forms and symbols in the dictionary. The normalized form is
the key of the dictionary and is added to the output, e.g., “euro” in the
example above or “cm” giving “centimetre”. The normalization makes the
comparison of different units easier. Note that conversions between metric
units is not supported. For example, “centimetre” is kept as the final
representation and not converted to “metre”.
If the detected surface form is shared across multiple _units_ , the unit is
_ambiguous_ and requires further normalization based on the context. Since
language models are great at capturing contextual information, for this
purpose, we train a BERT-based classifier Devlin et al. (2019). There are 18
ambiguous surface forms in our unit dictionary, and for each a separate
classifier is trained that allows to distinguish among _units_ based on the
context. If an ambiguous surface form is detected by the system, the relevant
classifier is used to find the correct normalized unit.
_Compound units_ are also detected and normalized independently. For example,
“kV/cm” results in “kilovolt per centimetre’, where. “kV” and “cm” are
normalized based on separate dictionary entries.

If no valid match in the dictionary exists, the surface form is tagged as a
_noun unit_ and lemmatizated, e.g., for “10 students” gives $u=student$. In
some cases, the adjective before a noun is also part of the unit, e.g., “two
residential suites” results in $u=residential\ suite$.

The _value dictionary_ contains the necessary information to standardize
_values_ to real numbers. More specifically, it contains surface forms for
prefixes and suffixes of scales, e.g., “B: billion” or “n: nano”, spelled out
numbers in textual format, e.g., “fourty-two: 42”, fractions in textual
format, e.g., “half: 1/2”, and scientific exponents, e.g., “$10^{2}$: 100’.
This combination is used to convert _values_ to decimal format. Scientific
notations with exponent and mantissa are converted to decimal values,
e.g.,“$2.3E2$ $\to$ $v=23$”.
Various trigger words or symbols for bounds and trends are managed in the
_changes dictionary_ , where detected tokens for _change_ are mapped to one of
the allowed categories $\sim,=,>,<,up,down$. For example, the entry for
equality is {"=": [ "exactly", "just", "equals", "totalling","="].

##  4 Evaluation

Table 1: Comparison of functionality for various extractors.

Feature |  Example |  CQE |  IllQ |  R-Txt |  Q3
---|---|---|---|---|---
Value |  5k euros (5k) |  ✓ |  ✓ |  ✓ |  ✓
Standardization |  5k euros (5000) |  ✓ |  ✓ |  ✓ |  ✓
Negative Values |  -5 C (-5) |  ✓ |  ✗ |  ✓ |  ✓
Fractions |  1/3 of the population (0.33) |  ✓ |  ✗ |  ✓ |  ✓
Range |  40-60 km/h (40-60) |  ✓ |  ✗ |  ✗ |  ✓
Non-quantities |  iPhone 11 (-) |  ✓ |  ✗ |  ✗ |  ✗
Scientific Notation |  $1.9\times 10^{2}$ (190) |  ✓ |  ✗ |  ✗ |  ✓
Unit |  1mm (mm) |  ✓ |  ✓ |  ✓ |  ✓
Unit normalization |  1mm (millimetre) |  ✓ |  ✗ |  ✓ |  ✓
Unit disambiguation |  10 pound (sterling or mass?) |  ✓ |  ✗ |  ✗ |  ✓
Noun Units |  200 people (people) |  ✓ |  ✓ |  ✗ |  ✗
Shared Units |  about 8 or $9 (both dollar) |  ✓ |  ✗ |  ✗ |  ✓
Change |  more than 100 (>) |  ✓ |  ✓ |  ✗ |  ✗
Trends |  DAX fell 2% (down) |  ✓ |  ✗ |  ✗ |  ✗
Concept |  AAPL rose 2% (AAPL) |  ✓ |  ✗ |  ✗ |  ✗

Table 2: Statistics of the number of sentences, quantities, and sentences with
and without quantities in the NewsQuant and R-Txt datasets.

Dataset |  #sent |  #quantity |  #sent with quantity |  #sent w/o quantity
---|---|---|---|---
NewsQuant |  590 |  904 |  475 |  115
R-Txt-currencies |  180 |  255 |  178 |  2
R-Txt-dimension |  93 |  121 |  77 |  14
R-Txt-temperature |  36 |  34 |  34 |  2
R-Txt-age |  19 |  22 |  18 |  1

CQE is compared against _Illinois Quantifier_ (IllQ), _Quantulum3_ (Q3),
_Recognizers-Text_ (R-Txt), and GPT-3 with few-shot learning Brown et al.
(2020). From here on, the abbreviation is used to refer to the respective
system. We first compare the functionality of the models, then describe our
benchmark dataset and compare the models on precision, recall and F1-score for
quantity extraction. Finally, unit disambiguation module is evaluated on a
custom-made dataset against Q3. Our evaluation code and datasets are available
at https://github.com/satya77/CQE_Evaluation.

###  4.1 Comparison of Functionality

Table 1 compares the functionality of the models in terms of different types
of _values_ , _units_ and _changes_ , as well as normalization techniques.
IllQ is the only baseline to detect _changes_ in _values_ but in a limited
setting that does not consider upward or downward trends. IllQ performs
normalization for currencies, however, _scientific units_ are not normalized.
Furthermore, it fails to detect _fractional values_. After our approach (CQE),
Q3 has the most functionality and is the only model that correctly detects
_ranges_ and _shared units_ and performs _unit_ disambiguation. On the other
hand, Q3 disregards _noun-based units_ , and although it is capable of
detecting a wide range of _value_ types, it makes incorrect detections of
_non-quantitative values_. R-Txt has dedicated models for certain quantity
types but fails to detect other types in the text. GPT-3 has a lot of
variability in the output and does not provide concrete and stable
functionality like the models discussed in this section, therefore, it is not
further considered in this comparison.

###  4.2 NewsQuant Dataset

For a quantitative comparison, we introduce a new evaluation resource called
NewsQuant, consisting of 590 sentences from news articles in the domains of
economics, sports, technology, cars, science, and companies. To the best of
our knowledge, this is the first comprehensive evaluation set introduced for
quantity extraction. Each sentence is tagged with one or more quantities
containing _value_ , _unit_ , _change_ , and _concept_ and is annotated by the
two first authors of the paper. Inter-annotator agreements separately for
_value_ , _unit_ , _change_ , and _concept_ is computed. For the first three,
the Cohen Kappa coefficient Cohen (1960) with values of $1.0$, $0.92$, and
$0.85$ is reported. Value detection is an easy task for humans and annotators
have perfect agreement. A _concept_ is a span of tokens in the text and does
not have a standardized representation, therefore, Cohen Kappa coefficient can
not be used and instead Krippendorff’s alpha Krippendorff (2011), with the
value of $0.79$, is reported. A set of guidelines was designed for what
constitutes a concept of higher quality. In total, the annotators completely
agreed on all elements for $62\%$ of the annotations.
We additionally evaluate four datasets available in the repository of R-Txt
for age, dimension, temperature, and
currencies444https://github.com/microsoft/Recognizers-
Text/tree/master/Specs/NumberWithUnit/English Last accessed: May 4, 2023.
These datasets contain only _unit/value_ pairs. The original datasets only
contained tags for the certain quantity type and would ignore other types,
giving the R-Txt model an advantage. For example, in the R-Txt-currencies,
only the currencies were annotated and other quantities were ignored. We added
extra annotations for all other types of quantities for a fair comparison. For
example, in the sentence “I want to earn $10000 in 3 years”, where only
“$10000” was annotated, we add “3 years”. Statistics of the number of
sentences and quantities for each dataset are shown in Table 2. The NewsQuant
dataset is the largest dataset for this task containing over 900 quantities of
various types. NewsQuant also includes negative examples with non-quantity
numerals.

###  4.3 Disambiguation Dataset

To train our unit disambiguation system, a dataset of 18 ambiguous surface
forms is created, using ChatGPT555https://chat.openai.com/ Last accessed: May
4, 2023. For each ambiguous surface form, at least 100 examples are generated,
and the final training dataset consists of 1,835 sentences with various
context information. For more challenging surface forms, more samples are
generated. For the list of ambiguous surface forms and the number of samples
for each class, refer to Appendix A.3. A test dataset is generated in the same
manner using ChatGPT, consisting of 180 samples, 10 samples per surface form.
For more information on the dataset creation, please see Appendix A.4.

Table 3: Precision, recall, and F1-score for detection of _value_ , _unit_ and
_change_ on NewsQuant.

Model |  Value |  Value+Unit |  Value+Change
---|---|---|---
P |  R |  F1 |  P |  R |  F1 |  P |  R |  F1
CQE |  $\mathbf{92.0}$ |  $\mathbf{91.9}$ |  $\mathbf{92.0}^{\dagger}$ |  $\mathbf{85.6}$ |  $\mathbf{85.5}$ |  $\mathbf{85.6}^{\dagger}$ |  $\mathbf{88.2}$ |  $\mathbf{88.1}$ |  $\mathbf{88.1}^{\dagger}$
Q3 |  65.0 |  83.3 |  73.0 |  42.1 |  53.9 |  47.2 |  - |  - |  -
IllQ |  50.6 |  66.0 |  57.3 |  32.8 |  42.8 |  37.1 |  44.2 |  57.6 |  50.0
R-Txt |  59.7 |  82.2 |  69.1 |  29.6 |  40.7 |  34.2 |  - |  - |  -
GPT-3 |  72.1 |  69.1 |  70.6 |  60.3 |  57.9 |  59.1 |  53.1 |  50.9 |  51.9

Table 4: Precision, recall and F1-score for detection of _value_ and _unit_ on
R-Txt Datasets.

Model | Detect | currency | dimension | temperature | age
---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
CQE | Value | $\mathbf{82.6}$ | 85.9 | $\mathbf{84.2}$ | $\mathbf{85.5}$ | 87.6 | $\mathbf{86.5}$ | $\mathbf{94.3}$ | 97.1 | 95.7 | $\mathbf{91.3}$ | $\mathbf{95.5}$ | $\mathbf{93.3}$
Q3 | 69.2 | 84.7 | 76.2 | 76.9 | $\mathbf{93.4}$ | 84.3 | 91.7 | 97.1 | 94.3 | $\mathbf{91.3}$ | $\mathbf{95.5}$ | $\mathbf{93.3}$
IllQ | 65.5 | 70.6 | 67.9 | 65.3 | 77.7 | 70.9 | 88.9 | 94.1 | 91.4 | 65.4 | 77.3 | 70.8
R-Txt | 67.4 | $\mathbf{91.8}$ | 77.7 | 73.6 | 90.1 | 81.0 | 91.9 | $\mathbf{100.0}$ | 95.8 | 77.8 | $\mathbf{95.5}$ | 85.7
GPT-3 | 50.5 | 54.9 | 52.6 | 80.2 | 80.2 | 80.2 | 93.5 | 85.3 | 89.2 | 92.3 | 54.5 | 68.6
CQE |  | $\mathbf{78.1}$ | $\mathbf{81.2}$ | $\mathbf{79.6}^{\dagger}$ | $\mathbf{78.2}$ | $\mathbf{80.2}$ | $\mathbf{79.2}$ | 91.4 | 94.1 | 92.8 | $\mathbf{91.3}$ | $\mathbf{95.5}$ | $\mathbf{93.3}$
Q3 | Value | 29.5 | 36.1 | 32.5 | 56.5 | 68.6 | 61.9 | 61.1 | 76.5 | 74.3 | 82.6 | 86.4 | 84.4
IllQ | +Unit | 41.8 | 41.6 | 45.1 | 43.4 | 52.1 | 47.5 | 30.6 | 32.4 | 31.4 | 42.3 | 50.0 | 45.8
R-Txt |  | 46.7 | 63.5 | 53.8 | 44.6 | 54.5 | 49.1 | $\mathbf{91.9}$ | $\mathbf{100.0}$ | $\mathbf{95.8}$ | 70.4 | 86.4 | 77.6
GPT-3 |  | 40.8 | 44.3 | 42.5 | 65.3 | 65.3 | 65.3 | 45.2 | 41.2 | 43.1 | 92.3 | 54.5 | 68.6

Table 5: Relaxed and strict matching, precision, recall and F1-score for
_concept_ detection on the NewsQuant.

Model | Relaxed Match | Strict Match
---|---|---
P | R | F1 | P | R | F1
CQE | $\mathbf{76.2}$ | $\mathbf{76.1}$ | $\mathbf{76.1^{\dagger}}$ | $\mathbf{57.0}$ | $\mathbf{57.0}$ | $\mathbf{57.0^{\dagger}}$
GPT-3 | 55.9 | 53.7 | 54.8 | 26.3 | 25.2 | 25.7

Table 6: Weighted micro-average precision, recall and F1-score on the _unit_
disambiguation dataset.

Model |  P |  R |  F1
---|---|---|---
CQE |  $\mathbf{89.9}$ |  $\mathbf{89.4}$ |  $\mathbf{88.1^{\dagger}}$
Q3 |  57.33 |  57.78 |  54.46

###  4.4 Implementation

CQE is implemented in Python 3.10. For dependency parsing, part-of-speech
tagging, and the matching of rules SpaCy 3.0.9666https://spacy.io/ Last
accessed: May 4, 2023 is used. The unit disambiguation module, with BERT-based
classifiers, is trained using spacy-
transformers777https://spacy.io/universe/project/spacy-transformers Last
accessed: May 4, 2023 for smooth intergeneration with other SpaCy modules.
Parsers were created to align the output format of different baselines so that
the differences in output representation do not affect the evaluation. For
instance, for IllQ, we normalize the _scientific units_ and account for
differences in the representation of ranges in Q3. If a value is detected by a
baseline but not standardized or a unit is not normalized to the form present
in the dataset, post-processing is applied for a unified output.
Moreover, to keep up with the recent trends in NLP and the lack of a baseline
for _concept_ detection, we introduce a GPT-3 baseline. The GPT-3 model is
prompted to tag quantities with 10 examples for few-shot learning. Prompts and
examples are available in our repository. We use the  text-davinci-003 model
from the GPT-3 API888https://platform.openai.com/ Last accessed: February 2,
2023 with a sequence length of 512, temperature of 0.5, and no frequency or
presence penalty. For more details, refer to Appendix A.2. We are aware that
with extensive fine-tuning and more training examples GPT-3 values are likely
to improve. However, the purpose of this paper is neither prompt engineering
nor designing training data for GPT-3, and the few-short learning should
suffice for a baseline.

###  4.5 Analysis of Results

All the models are compared on precision, recall, and F1-score for the
detection of _value_ , _unit_ , _change_ , and _concept_. Disambiguation
systems are also compared regarding precision, recall, and F1-score of unit
classification. Permutation re-sampling is used to test for significant
improvements in F1-scores Riezler and Maxwell (2005), which is more
statistically coherent in comparison to the commonly paired bootstrap sampling
Koehn (2004). Results denoted with $\dagger$ mark highly significant
improvements over the best-performing baseline with a $p$-value < 0.01.

NewsQuant: Table 3 shows the result on the NewsQuant dataset. Since Q3 and
R-Txt do not detect changes, respective entries are left empty. CQE beats all
baselines in each category by a significant margin, where most of the errors
are due to incorrect extraction of the dependency parsing tree and part-of-
speech tagging. The second best model, Q3, scores highly for _value_
detection, but misses all the noun base _units_ and tends to overgeneralize
tokens to _units_ where none exist, e.g., in “0.1 percent at 5884”, Q3,
detects _percent per ampere-turn_ as a _unit_. Moreover, Q3 makes mistakes on
different currencies and their normalization. We attribute this to their
incomplete unit dictionary. R-Txt works well for the quantity types with
dedicated models, but all the other quantities are ignored or misclassified.
IllQ has trouble with _compound units_ , e.g., “$2.1 per gallon” and tends to
tag the word after a _value_ as unit, e.g., in “women aged 25 to 54 grew by
1%”, _grew by_ is the detected _unit_. Although IllQ is supposed to normalize
currencies, in practice the normalization is limited and often currency
symbols are not normalized. Moreover, trends are ignored by IllQ, and the
model is biased to predict equality (_=_) for most _changes_ and other change
types are rare. GPT-3 achieves a score close to Q3 for the detection of
_units_ and _values_ and close to IllQ for _changes_. Nevertheless, due to
extreme hallucination, extensive post-processing of the output is required for
evaluation, e.g., many of the values extracted were not actual numbers and
units are not normalized. Moreover, GPT-3 often confuses value suffixes with
_units_ , e.g., “billion” or “million” and, despite the normalization prompt,
fails to normalize _units_ and required manual normalization for most
detections. Both IllQ and GPT-3 require extensive post-processing for units
and cannot easily be used outside of the box.

R-Txt Dataset: Evaluation results on the four quantity types of the R-Txt
dataset are shown in Table 4, where our model once again outperforms all
baselines on _value+unit_ detection for all categories except for temperature.
Nevertheless, for temperature, the R-Txt improvement over CQE is not
statistically significant. Small size of the age and temperature dataset
results in inconsistent significance testing. The closeness of _value_
detection between models is due to the structure of the dataset. _Value_ s are
floats, and the diversity of types like ranges, fractions, and non-quantities
is negligible. For more details in the error analysis on NewsQuant and R-Txt,
see Appendix A.6.

Concept Detection: Finally, _concept_ detection is evaluated on NewsQuant
dataset. Results are shown in Table 5. Following the approach of UzZaman et
al. UzZaman et al. (2013) for evaluation, strict and relaxed matches are
compared. A strict match is an exact token match, whereas a relaxed match is
counted when there is an overlap between the system’s and ground truth token
spans. Based on the scores we observe that _concept_ detection is harder in
comparison to _value+unit_ detection. Even GPT-3 struggles with accurate
predictions. Our algorithm for _concept_ detection is limited to common cases
and does not take into account the full complexity of human language, leaving
room for improvement in future work. Moreover, in many cases, the concept is
implicit and hard to distinguish even for human annotators. In general, our
approach is more recall oriented, trying to capture as many concepts as
possible, hence, the big gap between partial and complete matches. However,
since the method is rule-based, rules can be adjusted to be restrictive and
precision focused.

Unit Disambiguation: CQE is compared against Q3 (the only other systems with
disambiguation capabilities) in Table 6. Since the normalization of a _units_
is not consistent in the GPT-3 model and requires manual normalization, GPT-3
is left out of this study. All 18 classifiers are evaluated within a single
system. The results are averaged by weighting the score of each class label by
the number of true instances when calculating the average. CQE significantly
outperforms Q3 on all metrics, and it is easily expendable to new surface
forms and _units_ by adding a new classifier. Since the training data is
generated using ChatGPT, a new classifier can be trained using our paradigm
and data generation steps, as shown in in Appendix A.4. For a detailed
evaluation of each class, see Appendix A.5.

##  5 Conclusion and Ongoing Work

In this paper, we introduced CQE, a comprehensive quantity extractor for
unstructured text. Our system is not only significantly beating related
methods and frameworks as well as a GPT-3 neural model for the detection of
_values_ , _units_ and _changes_ but also introduces the novel task of
_concept_ detection. Furthermore, we present the first benchmark dataset for
the comprehensive evaluation of quantity extraction and make our code and data
available to the community. We are currently extending the extractor by
improving the quality of edge cases.

## References

  * Almasian et al. (2022) Satya Almasian, Milena Bruseva, and Michael Gertz. 2022.  Qfinder: A framework for quantity-centric ranking.  In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 \- 15, 2022_ , pages 3272–3277. ACM.
  * Banerjee et al. (2009) Somnath Banerjee, Soumen Chakrabarti, and Ganesh Ramakrishnan. 2009.  Learning to Rank for Quantity Consensus Queries.  In _Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009_ , pages 243–250. ACM.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2023) Sanxing Chen, Yongqiang Chen, and Börje F Karlsson. 2023.  Dataset and baseline system for multi-lingual extraction and normalization of temporal and numerical expressions.  _arXiv preprint arXiv:2303.18103_.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_ , 20(1):37–46.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Foppiano et al. (2019) Luca Foppiano, Laurent Romary, Masashi Ishii, and Mikiko Tanifuji. 2019.  Automatic identification and normalisation of physical measurements in scientific literature.  In _Proceedings of the ACM Symposium on Document Engineering 2019, Berlin, Germany, September 23-26, 2019_ , pages 24:1–24:4. ACM.
  * Forbus (1984) Kenneth D. Forbus. 1984.  Qualitative process theory.  _Artif. Intell._ , 24(1-3):85–168.
  * Ho et al. (2019) Vinh Thinh Ho, Yusra Ibrahim, Koninika Pal, Klaus Berberich, and Gerhard Weikum. 2019.  Qsearch: Answering quantity queries from text.  In _The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I_ , volume 11778 of _Lecture Notes in Computer Science_ , pages 237–257. Springer.
  * Huang et al. (2017) Wenhao Huang, Zijia Lin, Chris McConnell, and Börje F. Karlsson. 2017.  Recognizers-Text: Recognition and Resolution of Numbers, Units, and Date/time Entities Expressed Across Multiple Languages.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25-26 July 2004, Barcelona, Spain_ , pages 388–395. ACL.
  * Krippendorff (2011) Klaus Krippendorff. 2011.  Computing krippendorff’s alpha-reliability.
  * Lefrançois and Zimmermann (2018) Maxime Lefrançois and Antoine Zimmermann. 2018.  The unified code for units of measure in RDF: cdt: ucum and other UCUM datatypes.  In _The Semantic Web: ESWC 2018 Satellite Events - ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3-7, 2018, Revised Selected Papers_ , volume 11155 of _Lecture Notes in Computer Science_ , pages 196–201. Springer.
  * Li et al. (2021) Tongliang Li, Lei Fang, Jian-Guang Lou, Zhoujun Li, and Dongmei Zhang. 2021.  AnaSearch: Extract, Retrieve and Visualize Structured Results from Unstructured Text for Analytical Queries.  In _WSDM ’21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021_ , pages 906–909. ACM.
  * Maiya et al. (2015) Arun S. Maiya, Dale Visser, and Andrew Wan. 2015.  Mining Measured Information from Text.  In _Proceedings of the 38th International SIGIR Conference on Research and Development in Information Retrieval_ , pages 899–902. ACM.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Riezler and Maxwell (2005) Stefan Riezler and John T. Maxwell. 2005.  On some pitfalls in automatic evaluation and significance testing for MT.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 57–64, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Roy et al. (2015) Subhro Roy, Tim Vieira, and Dan Roth. 2015.  Reasoning about quantities in natural language.  _Transactions of the Association for Computational Linguistics_ , 3:1–13.
  * Sarawagi and Chakrabarti (2014) Sunita Sarawagi and Soumen Chakrabarti. 2014.  Open-domain Quantity Queries on Web Tables: Annotation, Response, and Consensus Models.  In _The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, New York, NY, USA - August 24 - 27, 2014_ , pages 711–720. ACM.
  * UzZaman et al. (2013) Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013.  Semeval-2013 task 1: Tempeval-3: Evaluating Time Expressions, Events, and Temporal Relations.  In _Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 1–9.

##  Appendix A Appendix

###  A.1 Value, Unit and Change Detection Rule

In this section, we provide two additional examples for _value_ , _unit_ , and
_change_ detection and describe the logic behind a few other rules.

Example 2: “The Meged field has produced in the past about 1 million barrels
of oil, but its last well was capped due to technical problems that have not
been resolved.”

  * •

NUM$\textunderscore$NUM detects the compound number of $1\ million$, where
$1$, a number, is the child of $million$, a noun, in the dependency tree.

  * •

QUANTMOD$\textunderscore$DIRECT$\textunderscore$NUM detects the relation
between the adjective “about” to the value $1$, which is later identified as
the change.

  * •

NOUN$\textunderscore$NUM$\textunderscore$ADP$\textunderscore$RIGHT$\textunderscore$NOUN
finds a noun or proper noun that has a number as a child in the dependency
tree. If there are prepositions in children of the noun, they are also
considered part of the unit. In this case, $[million,\ barrels,\ of,\ oil]$
are detected using this rule.

The naming of the rules is preserved in the repository. From the combination
of all rules, the candidate tokens $[about,\ 1,\ million,\ barrels,\ of,\
oil]$ are extracted as _value_ , _unit_ , and _change_.

Example 3: “They have a $3500 a month mortgage and two kids in private
school.”

  * •

NUM$\textunderscore$SYMBOL matches a symbol followed by a number. In this
case, $\$3500$ is detected.

  * •

NOUN$\textunderscore$NUM$\textunderscore$QUANT finds a number with a noun or
an adverb as its head in the dependency tree. Here, we have $[mortgage,\
3500,\ \$,\ month]$.

  * •

UNIT$\textunderscore$FRAC$\textunderscore$2 finds compound units with “per”,
“a” or “an” in between, e.g., $[3500,\ month,\ a]$

  * •

NOUN$\textunderscore$NUM detects a noun that has a number as a child, e.g.,
$[kids,\ two]$.

The mentioned rules contribute to extraction of two candidate quantities:
$[[\$,\ 3500,\ a,month,\ mortgage],[two,\ kids]]$.

###  A.2 GPT-3 and Few-shot Learning

To tag sentences using GPT-3, we use the few-shot learning paradigm by
prompting the model to tag quantities and units in the text, given 10 distinct
examples. GPT-3 is mainly advertised as a task-agnostic, few-shot learner, and
we have not performed extensive fine-tuning. With the 10 examples, we aim to
account for a variety of outputs, e.g., compound units, when no quantity is
present, noun-based units, and prefixes for scaling the magnitude of a value.
Our full prompt is as follows, where the quantities are outputted in a
numbered list, with an order of _change_ , _value_ , unit surface form, _unit_
, _concept_. The unit surface form is used in post-processing in case GPT-3 is
not able to normalize the unit.

    Tag quantities and units in the texts:

    Sentence: Woot is selling refurbished,
    unlocked iPhone XR phones with 64GB of
    storage for about $330.
    Answer:
    1. =, 1.64, GB, gigabyte, storage
    2. ~, 330, $, dollar, iPhone XR phones

    Sentence: The chain operates more than 600
    supermarkets and less than 800 convenience
    stores.
    Answer:
    1. >, 600, supermarkets, supermarkets,
    chain
    2. <, 800, convenience stores, convenience
    stores, chain

    Sentence: The spacecraft, which is about
    the size of a school bus, flew into Dimorphos
    at a speed of about 4.1 miles per second,
    that’s roughly 14,760
    miles per hour (23,760 kilometers per hour).
    Answer:
    1. ~, 1.4.1,  miles per second, mile per
    second, spacecraft
    2. ~, 14760, miles per hour, mile per hour,
    spacecraft
    3. ~, 23760, kilometers per hour, kilometer
    per hour, spacecraft

    Sentence: And overnight dogecoin fell from
    0.317 to 0.308, a 2.8 percent drop.
    Answer:
    1. =, 1.0.317-0.308, -, -, dogecoin
    2. =, 2.8, percent, percentage, dogecoin
    Sentence: This is about minus 387
    Fahrenheit (minus 233 Celsius).
    Answer:
    1. ~, -387, Fahrenheit, Fahrenheit, -
    2. ~, -233, Celsius, Celsius, -

    Sentence: WhatsApp more than 2 billion
    users send  fewer than 100bn messages a day.
    Answer:
    1. >, 2000000000, users, users, WhatsApp
    2. <, 100000000000, messages, messages,
    users

    Sentence: This includes colors between red
    and blue - wavelengths ranging between 390
    and 700 nm.
    Answer:
    1. =, 390-700, nm, nanometer, wavelengths

    Sentence: You don’t have a two-year
    bachelor’s degree or a six to eight-year
     phd degree.
    Answer:
    1. =, 2, year, year, bachelors degree
    2. =, 6-8, year, year, phd degree

    Sentence: The price CO2 and fuel
    consumption are not clear.
    Answer:
    No quantities or units

    Sentence:{sentence}
    Answer:

{sentence} is replaced with the query sentence to be tagged. Nevertheless, the
output of GPT-3 is not consistent and requires extreme post-processing. The
post-processing includes cleaning the predicted values to only include
numbers, normalization of the units even if the unit is miss-spelled, e.g.,
“celsiu” instead of “celsius”, “ppb” to “parts-per-billion”, or “€” to “euro”.

Table 7: Ambiguous surface forms, _units_ associate with them and the number
of samples in the training set for each surface form and unit pair.

Surface | Units | # samples
---|---|---
c | cent, celsius | 144
¥ | chinese yuan, japanese yen | 100
kn | croatian kuna, knot | 116
p | point, penny | 149
R | south african rand, roentgen | 100
b | barn, bit | 127
’ | foot, minute | 104
′ | foot, minute | 104
" | inch, second | 112
$"$ | inch, second | 112
C | celsius, coulomb | 116
F | fahrenheit, farad | 100
kt | kiloton, knot | 100
B | byte, bel | 107
P | poise, pixel | 102
dram | armenian dram, dram | 180
pound | pound sterling, pound-mass | 131
a | acre, year | 113

Table 8: Error analysis of different extraction systems.

Mistake | Systems
---|---
Trouble detecting temperature types, e.g., celsius and fahrenheit are both denoted as degree. | Q3, IllQ, GPT-3
Dollar types are not identified, e.g.,“hong kong dollar” and “new zealand dollar” $\to$ dollar. | Q3, IllQ
Unit normalization does not work for the majority of the times. | IllQ, GPT-3
Bias towards predicting $=$ for changes. | IllQ, GPT-3
Cryptocurrencies and rare ones are not recognized, e.g. Bitcoin or Markka. | Q3
Sports units are not recognized, e.g., ppg, rpg, apg. | Q3
Temporal values are mistaken as quantities, e.g., 2 pm. | Txt-R, GPT-3
Compound units are rarely found, e.g., kph. | Txt-R, GPT-3
Units in short sentences are not recognized, e.g., rmb 10 usd 20. | CQE
Problematic distinction between “year” and “year of age”. | CQE, Q3, GPT-3
Units are confused with concepts, e.g., building rate of $80 per sq m $\to$ “sq m” as a concept. | GPT-3
Low recall due to limited quantity types. | Txt-R
Detection of concepts where none exist. | CQE

###  A.3 Ambiguous Surface Forms

In our unit dictionary, we encountered 18 ambiguous surface forms with
different normalized units and collected at least 100 samples for each. This
list is not comprehensive and in different scientific domains, more ambiguous
cases might occur. The number of samples per surface form and associated units
for each surface form are shown in Table 7.

###  A.4 Disambiguation Prompts

To generate the dataset for disambiguation, we experimented with multiple
prompts, using ChatGPT. The aim was to create training/test data in JSON-
format, where the sentences are not duplicates or too simple. For this purpose
two sentences were formulated (one for each unit, in each surface form) and
are used as input examples of different contexts. The prompt explicitly asks
for JSON format output and 20 samples, due to the sequence length limitation
of ChatGPT. The final prompt is as follows, where UNIT1 and UNIT1 are replaced
with different units with the shared surface and "SURFACE_FORM" denotes the
ambiguous surface form:

    Create a training set of 20 samples, for
    "UNIT1" and "UNIT2", where in the text the
    surface form of the unit is always
    "SURFACE_FORM", but the unit is different.
    Output in JSON format as follows:

    {"text":"Sentence 1", "unit": "UNIT1" },
    {"text":"Sentence 2 ", "unit": "UNIT2" }}

The test dataset is created in the same manner. For certain units, multiple
generations were required to get more complex sentences. In such cases, we
specifically asked for sentences that do not start with "the" and are more
complex. After each generation, all examples were checked by the authors of
the paper. Faulty samples with wrong units were removed. In some cases,
surface forms were manually altered to match the specification of the task.

###  A.5 Disambiguation per Class

A detailed evaluation of the disambiguation dataset is shown in Table 9, where
precision, recall, and F1-score are computed separately for each class. For
each surface form, 10 examples are present in the test dataset. We noticed
that distinguishing between “japanese yen” and “chinese yuan ” is partially
difficult for the BERT-based classifier since both of them are currencies and
used in similar contexts. Another hard distinction is between “penny” and
“point”, since monetary values and the stock market point unit are used in
similar contexts. In comparison, in Q3 certain units are almost never
predicted, hence the multiple zeros in the evaluation results.

Table 9: Precision, recall and F1-score for _unit_ disambiguation per class.

Class | CQE | Q3
---|---|---
P | R | F1 | P | R | F1
knot | 100 | 100 | 100 | 16.67 | 100 | 28.57
roentgen | 100 | 100 | 100 | 44.44 | 80 | 57.14
barn | 100 | 80 | 88.89 | 100 | 80 | 88.89
japanese yen | 60 | 100 | 75 | 100 | 100 | 100
inch | 83.33 | 100 | 90.91 | 77.78 | 70 | 0.7368
armenian dram | 100 | 60 | 75 | 0 | 0 | 0
chinese yuan | 0 | 0 | 0 | 100 | 100 | 100
byte | 100 | 100 | 100 | 57.14 | 80 | 66.67
cent | 83.33 | 100 | 90.91 | 0 | 0 | 0
croatian kuna | 100 | 100 | 100 | 0 | 0 | 0
year | 100 | 100 | 100 | 0 | 0 | 0
poise | 100 | 100 | 100 | 100 | 100 | 100
south african rand | 100 | 100 | 100 | 100 | 60 | 75
minute | 80 | 80 | 80 | 60 | 90 | 72
bit | 83.33 | 100 | 90.91 | 50 | 40 | 44.44
bel | 80 | 100 | 88.89 | 100 | 100 | 100
kiloton | 100 | 100 | 100 | 100 | 100 | 100
second | 100 | 80 | 88.89 | 80 | 40 | 53.33
coulomb | 100 | 100 | 100 | 100 | 100 | 100
dram | 71.43 | 100 | 83.33 | 0 | 0 | 0
point | 55.56 | 100 | 71.43 | 0 | 0 | 0
fahrenheit | 100 | 100 | 100 | 100 | 100 | 100
celsius | 100 | 90 | 94.74 | 0 | 0 | 0
pixel | 100 | 100 | 100 | 80 | 80 | 80
pound-mass | 100 | 100 | 100 | 83.33 | 100 | 90.91
pound sterling | 100 | 100 | 100 | 100 | 80 | 88.89
foot | 80 | 80 | 80 | 100 | 50 | 66.67
penny | 100 | 20 | 33.33 | 0 | 0 | 0
farad | 100 | 80 | 88.89 | 80 | 80 | 80
acre | 100 | 100 | 100 | 0 | 0 | 0

###  A.6 Error Analysis on NewsQuant and Txt-R Datasets

We analyzed the incorrect detection for all the models and the common
mistakes. Except for the points discussed in the main part of the paper, Table
8 provides an overview of the remaining common mistakes and systems associated
with them.