missing

# Simplifying Electronic Document Digital Signatures

Lucas Mayr1, Frederico Schardong12, Ricardo Custódio1

###### Abstract

Electronic documents are typically signed using private keys and the matching
digital certificate through a Public Key Infrastructure (PKI). Private keys
must be kept in a safe place so they can be used multiple times. This makes
private key management a critical component of PKI for which there is no
foolproof answer. Existing solutions are often expensive and cumbersome. This
work proposes issuing an irrevocable digital certificate for each new document
to be signed. We demonstrate that an ephemeral key can be used with these
certificates, eliminating the need to store private keys. We analyze the
overhead caused by the requirement to generate new key pairs for each
document, provide a security overview and show the advantages over the
traditional model.

##  1 Introduction

Trusting cryptographic key management is one of the main challenges of public
key infrastructures (PKI) and their applications. Several cryptographic keys
are involved in PKIs. Digital certificates protect the public keys. On the
other hand, private keys are protected by cryptographic hardware or software
containers, which have access control mechanisms to the keys they are
responsible for.

The certificates are issued by Certification Authorities (CA). In addition to
public keys, CAs include data describing the key and its owner into the
certificates. The metadata is verified by Registration Authorities (RA) before
the CA issues the certificate at its holder’s request [2].

CAs and RAs generally employ Hardware Security Modules (HSMs) specifically
designed to ensure the creation and secure operation of their private keys. In
addition, these entities often operate in a safe room with many security
controls, making the operations even more secure. Of course, this
substantially increases the complexity and cost of CAs and RAs.

Even with all these precautions, CAs are still susceptible to cyber attacks.
For example, the DigiNotar incident showed that a process failure led to the
fraudulent issuance of certificates [4, 24]. That incident led to creating a
certificate enrollment protocol called Certificate Transparency to detect
incorrect issuance of certificates by CAs [12].

This new control further increased the complexity of PKIs. Many other controls
have been created since 1978 when Konfelder created the digital certificate
[11]. These mechanisms, however, have been the target of much criticism [7,
13, 8].

Managing end-user private keys, on the other hand, is more challenging. While
in a realistic PKI scenario we have a dozen CAs and RAs, the number of end
users can reach the millions. In addition, the operators of those entities are
usually specialized technicians and are accustomed to the complexity of the
devices used to protect the cryptographic keys. This is not the case for end
users. For end users, HSMs are prohibitively expensive and difficult to
handle.

As a result, end users use simpler, relatively low-cost cryptographic hardware
such as smart cards or crypto tokens. Recently, there has been a growing use
of cloud cryptographic key management service providers, preventing the user
from having to carry a smart card with them. However, using such cloud
services raises privacy concerns [16].

Another recurring problem in PKIs is the revocation of the certificates.
Revocation is necessary since certificates are issued to be valid for a
certain time. It is common to find end-user certificates valid for 1 or 3
years. CA certificates are issued for even longer periods, such as 10 or 25
years.

There are two methods generally used by CAs to publicize that a certificate
has been revoked. These are the certificate revocation list (CRL), and the
Online Certificate Status Protocol (OCSP) [22]. CRL is a blacklist; If a
certificate’s serial number is not in the CRL, then the certificate is
considered valid.

CRLs are challenging to manage. They are issued by CAs periodically and are
valid for short periods. The most significant difficulty is propagating the
CRL to all users so they can verify the validity of the public keys. Access to
CRLs is considered a major problem by applications and data communication
networks [17].

Because of this, big players ignored it by default, the most notable being TLS
in popular browsers, some checking a curated list of arguably important
revocations, and some checking OCSP queries.

The possibility of certificate revocation increases when we reach the end-user
level. Smart cards and tokens can be stolen, lost or malfunction. Cloud-based
solutions, for example, require the user to trust the cloud provider and
perform regular system audits.

Certificates can be used for many purposes. Perhaps the two biggest uses of
certificates are the authentication of Web servers using the TLS protocol
[19], and the digital signature of electronic documents. In this work, we
focus on this second application.

Deploying and managing the services of a PKI to issue certificates for users
to sign electronic documents is not straightforward. Although several
countries have had some success in implementing national PKIs, the vast
majority face the high cost of maintaining them. The cost of certificates is
often prohibitive for countries with limited resources.

Nowadays, all citizens should be able to sign electronic documents in their
daily lives. As we have seen, PKIs are not inclusive. Both from a knowledge
point of view and from an economic point of view. On the knowledge side, it
requires users to understand how PKI works and its complexities. Most notably,
it requires users to understand the importance of good key management and
revocation processes. For the average user, this is asking too much. On the
economic side, key management can become prohibitively expensive. In this work
we propose a new type of certificate and, consequently, an inclusive PKI.

For a PKI to be inclusive, it needs to eliminate the issues reported above.
PKI management and deployment, for example, should be simplified and cheaper.
Users shouldn’t worry about the intricacies of PKI or key management. For this
we propose unique certificates, that is, irrevocable certificates that use
ephemeral keys. We show that unique certificates solve the problem of key
management and complexity at the end-user level.

Contributions. The main contribution of this article is the introduction of
irrevocable Unique Certificates. Through these certificates, we can eliminate
the need for private key management and digital certificate revocation. We
compared our proposal with traditional digital certificates and PKI, showing
that while these ephemeral private keys present an overhead, we can circumvent
two of the worst challenges of traditional PKIs.

Structure of the article. This article is organized as follows. Section 2
presents the necessary background on PKI and digital certificates. In Section
3 we discussed the challenges of digital certificate revocation and private
key management and their impact on the widespread adoption of digital
signatures. Section 4 presents the Unique Certificates and presents our
proposal. In Section 5 we compare our proposal with the traditional method.
Finally, Section 6 presents our conclusions and future work.

##  2 Background

This section introduces the main aspects of PKI related to this work. Namely,
public key cryptography, hash functions, certification authorities and digital
certificates. We explore the need for digital certificates in the PKI
environment and briefly touch on their shortcomings.

###  2.1 Public Key Cryptography

Cryptography is the study and application of methods for securing information
so that an adversary cannot interfere with it without leaving traces.
Asymmetric Cryptography or Public-key cryptography is the name given to one
such cryptosystem. The principle behind asymmetric cryptography, which was
introduced in [6], is that it is possible to have two separate keys.
Typically, one of the keys is made public and the other is kept secret. The
published key is known as the public key and the one kept secret is called
private key. What is encrypted with one key can only be decrypted with the
other. This property allows public key cryptography to work in situations
where the data channel cannot be proven secure and removes the need for
parties to agree beforehand on a single secret key, as is the case with
symmetric cryptography.

Digital signature schemes work by combining a private key with a message to
produce an output that can be verified with a public key, as seen in Figure
1(b). Digital signatures were first proposed with the assumption that if a
one-way function existed, a signature scheme would follow [6]. Typically,
these schemes consist of three major steps: (i) key generation, which is
responsible for creating the key pair; (ii) message signing, and (iii)
signature verification, which should provide the following set of guarantees:

Authentication When the private key of a message is associated with a user,
verifying the document’s signature provides evidence that the document was
indeed signed by that user;

Integrity. Any alteration made to the document after the signature should
render the signature invalid;

Non-Repudiation. It is unlikely that the private key holder will need to deny
that he has signed the document.

###  2.2 Hash functions

Hash functions, a kind of one-way function, are easy to compute but
computationally difficult to invert. They receive an arbitrarily large message
and produce a fixed size output. Given the output of such a function, it
should be computationally difficult to determine what the input was. However,
generic one-way functions like these will not fulfill every demand.
Constraints were added to these functions to make them more useful.

A hash function is a one-way function formally described as
$F:\\{0,1\\}^{*}\rightarrow\\{0,1\\}^{n}$, where $n\in N$ is a fixed output
size [15]. Fixing the output of the one-way function to $n$ shrinks the co-
domain from $\\{0,1\\}^{*}$ to $\\{0,1\\}^{n}$. From this restriction, we can
see that since $\\{0,1\\}^{n}$ is a subset of $\\{0,1\\}^{*}$, hash functions
compress arbitrarily large inputs to a smaller image, therefore resulting in
unavoidable collisions. It is important to note that while collisions are
unavoidable by nature for hash functions, they should be unlikely to happen.

The concept behind hash functions is that the compressed value can serve as
the document’s fingerprint. Typically, digital signatures use hash functions
to reduce the size of the signature. The hash is signed instead of the entire
document, and the signature is appended to the document.

###  2.3 Digital Certificates

While public key cryptography allows participants to disregard the requirement
for a secure channel, published public keys must still be authenticated. The
initial solution to this issue was to utilize a trusted public file key
directory in which keys were associated with entities [6]. This solution had
many flaws derived mainly from its need for security and trust: (i) slow
operations; (ii) vulnerability to denial-of-service attacks; (iii) strong
incentives for attackers to breach the system [7].

To address these deficiencies, it was proposed that public keys should have
their lifecycle managed by a special infrastructure called Public Key
Infrastructure (PKI). In a PKI, the public keys are embedded in a file, called
a digital certificate. The digital certificate usually contains, in addition
to the public key, several other metadata that describes the type, use, and
identity of the holder of the private key. The entity responsible for issuing
these certificates is the Certification Authority (CA).

The use of digital certificates and CAs allowed for the separation of
signature verification and public key storage, enabling the replication of
certificate repositories that did not need to be secure or trusted [11]. Using
trusted CAs, these certificates effectively bind information to a public key
and ensure its validity. The process of generating a valid digital certificate
can be seen in Figure 1(a). X.509v3 is the current standard for digital
certificates, and minimal requirements are necessary for these certificates to
function [2]. These certificates can be adapted to a vast array of protocols
and applications by employing extensions.

To verify the validity of a digital certificate, its integrity and signatures
must be validated against a trusted CA. Additionally, we must confirm that the
certificates were valid when the signatures were created. A glaring issue must
be resolved within PKI: compromising a secret key could result in significant
damage. Therefore, there must be methods for validating the validity of a
certificate. This is typically achieved through the use of a certificate
revocation list (CRL). A CRL must contain a reference to every digital
certificate issued by a CA that has been revoked. Another popular alternative
to CRLs is the Online Certificate Status Protocol (OCSP), which responds to
queries about the status of a given certificate.

UserPrivate Key Public KeyCertificate Signing RequestCertificate
AuthorityCertificateGeneratesSignsIncluded InIssues (a) Certificate Issuance.

UserMessageHashSignaturePrivate Key Public KeyHashing
AlgorithmSignsGeneratesVerifies (b) Digital Signature.

Figure 1: Key pair operations.

The validity of a digital certificate is reliant upon two crucial operations.
One is to validate the chain of trust up to a trusted digital certificate.
This is accomplished by obtaining the digital certificate of the entity that
signed the certificate being validated and verifying its signature. This
process is repeated for each certificate being validated until a trusted
certificate is reached. The second step is to verify the revocation status of
each certificate. This can be accomplished in a variety of ways depending on
the infrastructure configuration. The most usual involves the download of a
CRL. Since CRLs are also signed documents, the user must also validate their
authenticity.

###  2.4 Certification Authority

Certificate Authority is the entity from which trust is derived. CAs work by
issuing digital certificates through the signing of information contained in a
Certificate Signing Request (CSR), which contains data about the certificate
holder along with their public key [2]. After verifying the signature of the
request, the certificate containing the requested data is issued by the CA.
The CA is ultimately responsible for attesting to the validity of the data
that is embedded in the certificate. Root CAs are entities the user chooses to
trust. These usually take the form of self-signed certificates. The root CAs
certificates are signed by the same entity which creates the CSR, effectively
attesting to their own information.

To verify the validity of a certificate, a user must travel along the
certification chain, validating each certificate until it reaches a root
certificate. CAs have the power and duty to revoke any certificate that has
had its information changed, the corresponding private key compromised, or
because the certificate holder has requested.

##  3 PKI Challenges

As we have seen, there are many challenges that need to be adequately
addressed for a PKI to be put into practice. There is consensus, however, that
the vast majority of problems are related to either certificate revocation or
to the management of the lifecycle of private keys. This section discusses and
details these challenges.

###  3.1 Private Key Management

Private key management is one of the main challenges of a PKI. Users are
identified and authenticated using digital certificates and private keys. CAs
rely on HSMs to store and use their private keys. End-users must rely on
either much simplified hardware or cloud-based solutions. Smart cards and
tokens are likely options for end-user hardware. For general document
signatures, cloud-based storage offers the advantage of not requiring a
physical device and is becoming increasingly common. However, cloud-based
solutions present a confidentiality risk since the private key is shared with
a third party. In their global encryption study for 2021, Entrust discovered
that 56 percent of respondents rated key management as extremely difficult
[20]. Below are the main devices used to manage private keys in PKI and their
related shortcomings.

HSM. HSMs are tamper-resistant hardware devices that protect cryptographic
operations by generating, protecting, and managing the private keys required
for encrypting and decrypting data, for creating digital signatures and
certificates. To test, validate, and certify HSMs, the highest security
standards, such as FIPS 140-2 and Common Criteria, are utilized [1, 5]. Used
almost exclusively by CAs, HSMs provide the best possible protection for
private keys, but are also the most expensive and complicated solution. This
approach is impractical for end users due to prohibitive pricing and
management requirements.

Cloud. The advent of cloud computing introduced another option for key
management. Key Management as a Service (KMaaS) emerged as a great option to
remove the need for users to carry around any type of hardware themselves.
These services usually offer their own HSMs to users. The delegation of key
management to a third party introduces security concerns. Most notable, it
brings a lack of confidentiality. The emergence of cloud-based solutions to
the problem of private key management has been essential to the growth of
digital signatures for the general public [3]. However, it assumes the general
public knows enough to keep constant watch over its private key and perform
periodic audits. The compromise of the private key also requires the user to
have some knowledge of the revocation process. These requirements are often
neglected and are not realistic when considering the widespread use of digital
signatures.

Smart Card. Smart Cards are plastic cards the size of a credit card containing
an embedded memory and microprocessor. These cards have enough computational
power to handle simple tasks and are used in many places in the industry.
Smart cards used in PKI utilize the PKCS#15 structure to store user
credentials such as keys, certificates and passwords [18]. These cards have
been used extensively in PKI and are one of the most common ways to manage
private keys [14]. Although they are one of the oldest ways to manage private
keys, these devices remain expensive and complex as they require additional
hardware known as smart card readers.

Token. Tokens are similar to smart cards. They also vary in price depending on
their specifications, but are generally a bit pricier than smart cards. The
biggest difference is that tokens combine smart card and reader in a single
device, simplifying operation and improving usability. However, tokens do not
have any intrinsic advantages or new functions when compared to smart cards.
Tokens have a secondary chip dedicated to translating USB messages to ISO7816.
This secondary chip can be the target of attacks since it acts as the middle-
man privy to sensitive information.

###  3.2 Certificate Revocation

PKI was developed to guarantee the validity of asymmetric encryption [11]. The
identity of the holder of the private key is determined by the private key
corresponding to a digital certificate issued by a trusted chain of
certificates. Therefore, these private keys should be stored and utilized with
due caution. However, the system should be able to handle situations in which
this private key has been compromised and swiftly revoke it. PKI handles such
scenarios by revoking the issued digital certificates and declaring them
invalid. One of the issues with working with PKIs is that certification does
not handle the compromise of private keys well [11]. The two most suitable
solutions to address the certificate revocation problem are based on
certificate revocation list (CRL) and online certificate status protocol
(OCSP). However, none of them is a perfect solution to these issues[7].

CRLs are lists that contain the serial number of certificates that have been
revoked by a certificate authority. Depending on the objective of the CA,
these lists have validity periods ranging from hours to months. CRLs are
signed by the CA. These lists have numerous disadvantages:

  * •

New CRLs must be issued even if no new certificates have been revoked;

  * •

Compromised private keys can be utilized until the new CRL is issued;

  * •

CRLs can grow to large file sizes;

  * •

Network load spikes when the CRL expires;

  * •

CRL URLs can become unreachable;

  * •

Human error can cause certificates not to be revoked or to be incorrectly
added to the CRL.

Changes have been made to the way CRLs work to tackle some of these problems.
Redundant URL endpoints can be used to lessen the impact of unreachable URLs,
Delta CRL is a variant that aims to lessen the impact of having long periods
between updates by frequently publishing smaller CRLs that complement the
regular one.

Another popular revocation approach, OCSP, and its variant, OCSP Stapling,
provide a solution to the problem of huge CRL sizes affecting clients [22].
When it is necessary to look up the revocation status of a digital
certificate, an OCSP request is sent to the OCSP server, which checks the
status of the certificate, usually using a local CRL and returns a response of
valid, revoked, or unknown. This form of revocation eliminates the issue of
redundancy and significantly reduces the impact of big CRLs and the time
required for the revocation to take effect. However, due to its online nature,
it is vulnerable to denial-of-service attacks. Additionally, even while it can
revoke certificates relatively quickly, it can only do so after the key
compromise has been detected.

There have been numerous attempts to resolve the problem of certificate
revocation. Nevertheless, it appears that none of the proposed solutions are
satisfactory, and we have accepted certificate revocation as an unavoidable
aspect of PKI, deeming CRL and OCSP ”good enough” solutions for the time
being. Given how much power the private key is granted and how dangerous it is
to have a compromised key for extended periods, it is clear that efficient,
rapid, and straightforward revocation is necessary. We propose that the first
step in resolving certificate revocation is to lower the amount of damage a
compromised private key can cause, so that the end user no longer has to
safeguard it. Our proposal is described in the next section.

##  4 One-Time Digital Certificates

Certificate revocation is required due to the risks associated with
unauthorized use of a private key, often at the end-user level. Smart cards,
tokens, and cloud-based solutions are the primary methods for securing private
keys at this level. Smartcards and tokens are very similar in that they are
both physical devices that must be safeguarded and actively used to access the
private key. These devices can be misplaced or improperly secured, especially
by non-technical users who lack an understanding of the gravity of compromised
private keys. Cloud solutions eliminate the risk of losing a physical device;
however, the end-user must still be aware of the dangers a compromised private
key can cause and should audit its key on a periodic basis; this again
presents a barrier to the widespread adoption of digital certificates among
non-technical users. Private key management and PKI as a whole must be
transparent to users for an inclusive PKI to be widely used and secure.

The concept of irrevocable certificates is not new; it has been suggested by
Simple Public Key Infrastructure (SPKI) as an alternative form of
certification; nonetheless, it is deemed too risky for widespread use as is.
Rivest [21] proposes that a certificate’s own recency is the strongest
evidence of its legitimacy [21]. Nevertheless, we must still declare a private
key compromised when the need arises. Rivest addresses this problem by
separating key compromise from CAs, where it is handled by a key compromise
agent or suicide bureau, transferring revocation from certificate to private
key directly, and introducing a new service that must be verified prior to
accepting a certificate.

For an inclusive PKI to be widely adopted and secure, end-users should not be
expected to understand private keys or even PKI itself. The entirety of
private key management and PKI should be transparent to end users. This means
eliminating the need for any hardware such as smart cards or tokens, as well
as the inherent confidentiality loss associated with keeping private keys in
the cloud.

We introduce One-Time Certificates (OTCs), long-term irrevocable digital
certificates bound to a single document that use ephemeral keys. Firstly, OTCs
use ephemeral keys to remove the need for key storage. Key pairs are created
to sign a CSR and a document and subsequently deleted. Secondly, the hash of
the signed document is added to a X.509 extension in the CSR and the
corresponding OTC. Figure 2 illustrates OTCs. OTCs can only validate a
signature corresponding to the hash contained within. This binding restricts
the use of the private key to a single signature. In this way, the compromise
of the private key would be meaningless, as it could only be used to generate
the same signature that has already been generated.

Our new PKI has a path of certificates, from the root certificate to the end
user certificate, all with the same validity time. For example, if a root
certificate is worth 25 years, then certificates from all CAs and also from
the end user will have the same validity time.

UserPrivate Key Public KeyCertificate Signing RequestHashCertificate
AuthorityCertificateGenerate Key PairSignsIncluded InIssuesIncluded In Figure
2: One-Time Certificate Issuance Overview.

Due to the irrevocable nature of OTCs, issuer CAs are not required to issue
CRLs beyond a single blank CRL to ensure compatibility with existing and
legacy applications. The elimination of the need for CRLs from CAs relieves
them of the need of backing up their private keys, allowing these keys to be
marked as non-exportable in HSMs. Revocation of an issuer CA can be
accomplished by deleting the appropriate private key. Since issuer CAs below
the same intermediate CA are functionally identical so long as they employ the
same algorithms, an intermediate CA can issue as many issuer CAs as necessary
for load balancing.

##  5 Discussion

In this section, we compare our proposal to the structure and operation of
traditional PKIs. The implications of disposable private keys on certificate
revocation and private key management are discussed. In addition, we
demonstrate that security is either maintained or enhanced through the use of
OTCs.

###  5.1 Security Overview

One of the most important requirements for widespread adoption of new
technology is that it requires as little of its users as possible. In the case
of digital signatures for documents, this entails understanding the importance
of keeping their private key safe, what could happen if it were compromised,
and how to detect and respond to such compromises, which is quite a tall
order. The destruction of private keys and the binding of the document to the
certificate effectively reduces the private key’s impact in PKI, allowing us
to either reduce or eliminate many of these user requirements. First, let us
compare the traditional use of a private key for document signing, where the
key is kept safe (Case I), with another where the private key is discarded
after signing the first document (Case II).

I. The user activates its private key that has been stored in a secure device
and signs a document. The private key must be kept secure until the end of the
certificate lifetime. To sign a new document, the user activates its private
key once again and signs. In the case of a compromise, the attacker can sign
any document, any number of times. The user must request the certificate to be
revoked.

II. The user generates an ephemeral key pair and signs the document. The
private key is destroyed. The private key must be kept secure for a brief time
between key pair generation and document signature. To sign a new document,
the user must generate another key pair and request a new certificate. In the
case of a compromise, the attacker can sign any document, any number of times.
The user must request the certificate to be revoked.

Comparing cases I and II, we can see two major changes. Firstly, to sign a new
document, the user in case II must generate another key pair and request a new
certificate, similar to OTCs. Secondly, the private key must be protected for
a shorter period, restricting the time-frame an attacker would have to steal
it. However, were the key to be compromised in this period, the consequences
would be the same for both cases. Let us now compare the same experiment while
using OTCs, where one user refuses to delete their key (Case IV) and another
does (Case III).

III The user generates an ephemeral key pair and signs the document. The
private key is destroyed. The private key should be kept secure for a brief
time between key pair generation and document signature. To sign another
document, the user must generate another key pair and request a new OTC. The
OTC can not be revoked. The acceptor can request a more recent OTC to
guarantee the information is fresh. An attacker can sign the same document
that was already signed any number of times.

IV. The user signs the document using its private key and stores it in a
secure device. The private key should be kept secure while not being
destroyed. To sign another document, the user activates its old private key,
but must request a new OTC. The OTC can not be revoked. The acceptor can
request a more recent OTC to guarantee the information is fresh. An attacker
can sign the same document that was already signed any number of times.

The first important change from the traditional signature requirements is that
the requirement of private key storage has been lessened. Binding a document
to a key pair allows us to limit the potential dangers the compromise a
private key can achieve. Key compromises in OTCs are less severe. Acceptors
that verify the hash of the document will only accept documents that have
already been signed by the original user. The second important information
extracted from these examples is that the disposable nature of OTC private
keys is not a hard requirement. Users can opt to keep their keys secure
through the same means as traditional PKI or to sign any number of documents
before destroying the private key. The recommendation of private key
destruction comes from the possibility of simplifying the PKI structure for
its widespread use, but it can be flexible.

###  5.2 Irrevocable Certificates

Revocation is a statement from the CA that declares that certain certificates
are revoked. A certificate could be set for revocation, but it will only still
be valid sometime after the revocation request. These can be thought of as
recency requirements issued by CAs. A certificate with a six month CRL period
has its validity renovated every six months. It was proposed that these
recency requirements should not be set by the CA, but by the acceptors of
certificates [21]. The characteristics of OTCs are suited for this shift,
eliminating revocation from these certificates and leaving recency
requirements in the hands of the recipients. We can see from comparing Figure
1(a) and Figure 2 that the only major difference is the inclusion of a
document hash into the CSR and eventually into the certificate.

The idea of irrevocable certificates is not new. It has been proposed a few
times [25, 21]. However, none of these proposals could guarantee the safety of
the user. The main problem was that the private key could be used to sign any
number of documents while its certificate was valid. Without any type of
revocation, gains from its removal were too little to justify. By binding
private key-document-certificate, we can forgo these worries and create
irrevocable certificates in a safe manner. While the certificates themselves
are irrevocable, an acceptor could define a recency requirement and accept
only certificates issued within a suitable time-frame.

###  5.3 Deployment Costs

Comparing the deployment costs of both OTC PKI and traditional PKI, we can see
that the two aspects that have changed are user key management and issuer CAs.
Let $C$ be the cost traditional and OTC PKI share. OTC issuer CAs will
inevitably sign a higher number of certificates, introducing a cost overhead
$\delta$. Every user on traditional PKI must employ some key management
solution that has an associated cost $C$. A PKI with $n$ users will spend
$n*C$ for key management. OTC PKI reduces the key management cost to zero and
for a large enough PKI, we have that $\delta$ is negligible when compared to
$C$. This leads us into the deployment costs show in Table 1 that shows that
by using OTC PKI we can lower the deployment cost by $n*C$.

Table 1: Deployment Cost Comparison. | CA Management | Key Management | Total Cost
---|---|---|---
OTC PKI | $S+\delta$ | $0$ | $S+\delta$
Traditional PKI | $S$ | $n*C$ | $S+n*C$

Consider the following hypothetical scenario. A large country wants to
implement a PKI for document signing for its population. This country has a
population of 100 million. They want to avoid the confidentiality risks of
cloud-based solutions and decide that each person will have its own key
management device. Suppose each device costs about US$1, resulting in an
upfront cost of US$100 million. This cost could be entirely avoided by using
an OTC PKI.

###  5.4 OTC Issuance Overhead

By issuing new certificates for each signature, we impose a considerable
amount of overhead to the signing process. Using performance comparisons [10],
we can determine the practical impact of our proposal. Traditional schemes
favor signature speed above key generation speed. However, as shown in Table 2
and Table 3, our model works best when adopting signature schemes with a low
sum of signature plus key generation times.

Table 2: RSA Overhead for OTCs. Key length (bits) | Key Generation (s) | Signature Time (s) | Total Time (s) | Overhead
---|---|---|---|---
1024 | 0.16 | 0.01 | 0.17 | 1600%
2240 | 7.47 | 0.15 | 7.62 | 4980%
3072 | 9.80 | 0.21 | 10.01 | 4667%
7680 | 133.90 | 1.53 | 135.43 | 8752%
15360 | 679.06 | 9.20 | 688.26 | 7381%
Table 3: ECDSA Overhead for OTCs. Key length (bits) | Key Generation (s) | Signature Time (s) | Total Time (s) | Overhead
---|---|---|---|---
163 | 0.08 | 0.15 | 0.23 | 53%
233 | 0.18 | 0.34 | 0.52 | 53%
283 | 0.27 | 0.59 | 0.86 | 46%
409 | 0.64 | 1.18 | 1.82 | 54%
571 | 1.44 | 3.07 | 4.51 | 47%

###  5.5 Final Remarks

We can avoid a number of difficult-to-solve problems by constructing a PKI
with these modifications, which are summarized in Table 4. By associating a
document and key pair with a digital certificate, OTCs reduce the damage that
a compromised key can cause and the knowledge required to use digital
signatures securely. This binding also enables PKI to issue irrevocable
certificates, eliminating the need for revocation. Additionally, these
irrevocable certificates provide acceptors with more control when deciding
which certificates to accept. An unfavorable consequence is the need to issue
new certificates for each signed document; this consequence becomes more
severe the more frequently a user signs new documents.

As a result, OTCs are best suited for environments where a large number of
users wish to sign documents only a few times and who either do not care about
key management or do not wish to worry about it. OTCs also lower the
deployment cost of PKI. Large PKI environments benefit from having no cost
related to end-user hardware, be it direct costs or management.

Table 4: Comparison between traditional certificates and OTCs. Characteristic | Traditional | OTC
---|---|---
Signature Overhead | No | Yes
Revocation | Yes | No
Key Reuse | Yes | No
Key Management Hardware | Yes | No
Key Compromise Impact | High | Low
Deployment Costs | High | Low

##  6 Conclusion

The need for user-level key management is a problem that must be resolved
before digital signatures can be widely adopted by the general public. The
removal of hardware requirements for these users due to cloud solutions
increased the adoption of digital signatures. However, it also introduces some
inherent cloud computing problems, such as confidentiality concerns. By
relieving the user of all key management responsibilities, we can safely
sidestep this restriction.

Along with improving usability for end-users, OTCs lower the general costs of
PKI. The removal of hardware means that any cost associated to them is removed
as well. This means no cost associated with the management of hardware,
logistics and the hardware itself. This is a crucial change to enable large
PKIs to flourish in countries where there is a need for cheap and simple PKI.

Certificate revocation is another challenging PKI problem. There are
deficiencies in the existing schemes, as presented earlier. It has been
suggested that the optimal solution for revocation is to remove it entirely.
The core problem with long-term, irrevocable certificates is the risk of
damage caused by a compromised private key. The binding of a document to a
certificate solves the core issue with these irrevocable certificates, as it
reduces the amount of damage a compromised private key can cause and enables
PKI to forego user-level revocation. Backward compatibility is easily achieved
by using a single long-term blank CRL.

The disposable nature of private keys in OTCs allows us to consider using
other signature methods, such as one-time signature schemes. These algorithms
can only sign a finite number of documents, but can have some desirable
characteristics. As an example, hash-based signature schemes are often simple,
efficient, and resistant to quantum computer attacks [9]. The finite nature of
these schemes is irrelevant to OTCs, since every OTC has a corresponding
document. These types of signatures already seem to be a strong contender for
post-quantum internet of things signatures [23] and could be used in tandem
with OTCs.

Removing revocation from user certificates is an essential first step in
developing a PKI that eliminates revocation altogether. Building a PKI without
revocation in its entire chain using OTCs may appear daunting, but it can be
accomplished by using OTCs and the effect it has in the certification chain.

## References

  * [1] Elaine Barker, Elaine Barker, William Burr, William Polk, Miles Smid, et al.  Recommendation for key management: Part 1: General.  National Institute of Standards and Technology, Technology Administration …, 2006.
  * [2] Sharon Boeyen, Stefan Santesson, Tim Polk, Russ Housley, Stephen Farrell, and David Cooper.  Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile.  RFC 5280, May 2008.
  * [3] Kyung Won Chong, Yong Seok Kim, and Jeongil Choi.  A study of factors affecting intention to adopt a cloud-based digital signature service.  Information, 12(2):60, 2021.
  * [4] Comodo.  Comodo fraud incident 2011-03-23, 2011.  Available at: https://web.archive.org/web/20220423170320/https://www.comodo.com/Comodo-Fraud-Incident-2011-03-23.html. Acessed on 2022-05-16.
  * [5] Michael J Cooper, Kim B Schaffer, et al.  Security requirements for cryptographic modules.  2019\.
  * [6] Whitfield Diffie and Martin Hellman.  New directions in cryptography.  IEEE transactions on Information Theory, 22(6):644–654, 1976.
  * [7] P. Gutmann.  Pki: it’s not dead, just resting.  Computer, 35(8):41–49, 2002.
  * [8] Hilda Hadan, Nicolas Serrano, and L Jean Camp.  A holistic analysis of web-based public key infrastructure failures: comparing experts’ perceptions and real-world incidents.  Journal of Cybersecurity, 7(1):tyab025, 2021.
  * [9] Andreas Huelsing, Denis Butin, Stefan-Lukas Gazdag, Joost Rijneveld, and Aziz Mohaisen.  XMSS: eXtended Merkle Signature Scheme.  RFC 8391, May 2018.
  * [10] Nicholas Jansma and Brandon Arrendondo.  Performance comparison of elliptic curve and rsa digital signatures.  nicj. net/files, 2004.
  * [11] Loren M Kohnfelder.  Towards a practical public-key cryptosystem. PhD thesis, Massachusetts Institute of Technology, 1978.
  * [12] Ben Laurie.  Certificate transparency.  Communications of the ACM, 57(10):40–46, 2014.
  * [13] Alexander Lechner.  Low probability, high stakes: A look at pki.  Innovative Internet Technologies and Mobile Communications (IITM), 11, 2012.
  * [14] Eric Longo and Jeff Stapleton.  Pki note: Smart cards.  In PKI Note Series, PKI Forum, 2002.
  * [15] Ralph C Merkle.  A certified digital signature.  In Conference on the Theory and Application of Cryptology, pages 218–238. Springer, 1989.
  * [16] Muhammad Faheem Mushtaq, Urooj Akram, Irfan Khan, Sundas Naqeeb Khan, Asim Shahzad, and Arif Ullah.  Cloud computing environment and security challenges: A review.  International Journal of Advanced Computer Science and Applications, 8(10), 2017.
  * [17] Moni Naor and Kobbi Nissim.  Certificate revocation and certificate update.  IEEE Journal on selected areas in communications, 18(4):561–570, 2000.
  * [18] Magnus Nyström.  Pkcs# 15-a cryptographic token information format standard.  In Smartcard, 1999.
  * [19] Yngve N. Pettersen.  The Transport Layer Security (TLS) Multiple Certificate Status Request Extension.  RFC 6961, June 2013.
  * [20] Poneon Institute.  2021 global encryption trends study.  Technical report, Poneon Institute, 2021.  Available at: https://web.archive.org/web/20220531133514/https://images.go.entrust.com/Web/EntrustInc/%7Ba72c4c3a-f0cc-4542-8344-f070caf24d07%7D_global-encryption-trends-study-re.pdf (accessed on 31 May 2022).
  * [21] Ronald L Rivest.  Can we eliminate certificate revocation lists?  In International Conference on Financial Cryptography, pages 178–183. Springer, 1998.
  * [22] Stefan Santesson, Michael Myers, Rich Ankney, Ambarish Malpani, Slava Galperin, and Dr. Carlisle Adams.  X.509 Internet Public Key Infrastructure Online Certificate Status Protocol - OCSP.  RFC 6960, June 2013.
  * [23] Sabah Suhail, Rasheed Hussain, Abid Khan, and Choong Seon Hong.  On the role of hash-based signatures in quantum-safe internet of things: Current solutions and future directions.  IEEE Internet of Things Journal, 8(1):1–17, 2020.
  * [24] Nicole Van der Meulen.  Diginotar: Dissecting the first dutch digital disaster.  Journal of strategic security, 6(2):46–58, 2013.
  * [25] Tatu Ylonen, Brian Thomas, Butler Lampson, Carl Ellison, Ronald L. Rivest, and William S. Frantz.  SPKI Certificate Theory.  RFC 2693, September 1999.


# Specifying Genericity through Inclusiveness and Abstractness Continuous
Scales

###### Abstract

This paper introduces a novel annotation framework for the fine-grained
modeling of Noun Phrases’ (NPs) genericity in natural language. The framework
is designed to be simple and intuitive, making it accessible to non-expert
annotators and suitable for crowd-sourced tasks. Drawing from theoretical and
cognitive literature on genericity, this framework is grounded in established
linguistic theory. Through a pilot study, we created a small but crucial
annotated dataset of 324 sentences, serving as a foundation for future
research. To validate our approach, we conducted an evaluation comparing our
continuous annotations with existing binary annotations on the same dataset,
demonstrating the framework’s effectiveness in capturing nuanced aspects of
genericity. Our work offers a practical resource for linguists, providing a
first annotated dataset and an annotation scheme designed to build real-
language datasets that can be used in studies on the semantics of genericity,
and NLP practitioners, contributing to the development of commonsense
knowledge repositories valuable in enhancing various NLP applications.

Keywords: Collaborative Resource Construction and Crowdsourcing, Corpus,
Semantics

Specifying Genericity through Inclusiveness and Abstractness Continuous Scales

Claudia Collacciani, Andrea Amelio Ravelli, Marianna Marcella Bolognesi
---
Faculty of Modern Languages, Literatures and Cultures
University of Bologna, Bologna, Italy
{claudia.collacciani2, andreaamelio.ravelli<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Language allows us to convey both information about particular individuals and
situations, as in (1a), and generalizations about kinds, as in (1b).

  1. 1.

    1. (a)

The lion escaped yesterday from the zoo.

    2. (b)

The lion is a predatory cat.

The same noun phrase NP ("The lion" in the examples above) can be used in both
interpretations. The syntactic form of the NP (definite, indefinite, plural)
is not sufficient to disambiguate between the two meanings: the disambiguation
is guided by the context in which the NP occurs Krifka et al. (1995). This
phenomenon can be found in every language Behrens (2005) and in virtually all
lexical items that can be employed in referring expressions (i.e., nouns).
Nevertheless, there is no explicit marker for generic NPs in natural languages
Dahl (1995): the NPs’ genericity is determined by the meaning of the sentence
as a whole.

Statements about kinds, such as (1b), defined generics, can be seen as
fundamental to human cognition, because they enable us to conceptualize
properties associated with categories, structuring our perception of the world
Chatzigoga (2019).

Existing annotation frameworks capture levels of genericity in linguistic
expressions using discrete multi-class annotation schemes (Mitchell et al.,
2003; Walker et al., 2006; Friedrich et al., 2015) or continuous multi-label
systems Govindarajan et al. (2019). The use of such systems can be appropriate
for cases such as those in (1a) and (1b), where the subject clearly refers
respectively to a specific individual and to a category, and it is
straightforward to assign each to the generic or non-generic meaning. However,
the expression of genericity and its perception in speakers’ minds is a very
complex semantic aspect, which cannot be fully modeled by classifications of
this kind. In fact, this type of annotations can denote a noun in context as
generic or not (or even as both simultaneously, in the case of non-exclusive
labels, such as those in Govindarajan et al., 2019), but never specify how
much or in what way it is generic. In contrast, recent works in the cognitive
domain tend to represent the expression and perception of genericity through
continuous models Tessler and Goodman (2019): as we will see, not all generics
are the same.

The most distinctive feature of generics is that they allow for exceptions
Krifka et al. (1995), enabling speakers to interpret their quantification
value in different ways relying on their world knowledge. This is shown from
contrasts such as that between (2a) and (2b).

  1. 2.

    1. (a)

Robins are birds.

    2. (b)

Robins lay eggs.

Both the statements in (2a) and (2b) are true and the NP subject is generic in
both cases. However, in the first case it refers to all the individuals of the
category (all robins are birds), while in the second it refers to only some of
them (only adult females lay eggs).

Furthermore, as illustrated by the taxonomic reference phenomenon Carlson and
Pelletier (1995), the same NP can refer not only to an individual and to a
category, as in (3a) and (3b) respectively, but also to a subcategory, as in
(3c).

  1. 3.

    1. (a)

A whale dove back into the ocean.

    2. (b)

A whale is a marine mammal.

    3. (c)

A whale which was recently put under protection is the blue whale.

Another distinction is that between characterizing generics and direct kind
predications Carlson (1977); Leslie and Lerner (2022). The first ones are
statements such as (4a), which predicate applies to individual members of the
kind; the second ones are sentences such as (4b), which predicate cannot refer
to an individual, but only to a kind.

  1. 4.

    1. (a)

Tigers are striped.

    2. (b)

Tigers are widespread.

It’s worth noting that the indefinite singular form can be used in
characterizing generics but not in direct kind predications: the sentence *A
tiger is widespread is not felicitous 111Sentences such as this can be used
only if the NP ’A tiger’ is construed as a case of taxonomic reference, i.e.,
as referring to a species of tiger, thus still to the kind and not to the
individual..

We also want to point out that the literature on genericity has hardly dealt
explicitly with abstract nouns, tending to present the contrast between
generic and non-generic meaning almost always through concrete nouns, as we
also did in the examples given so far. This is probably the case because,
intuitively, it appears that distinguishing between generic and non-generic
meaning for abstract entities is less straightforward than for concrete ones.

We propose a novel annotation framework to model the expression of NPs’
genericity in a fine-grained manner, that is both grounded in linguistic
theory and intuitive enough to be carried out by crowdsourcing.

Our purpose is twofold. On one hand, we are interested in investigating if,
from naive language users annotations, differences emerge that trace back to
phenomena such as the ones above, observed in theoretical literature by
experts (we carry out this investigation particularly in §5.2). On the other
hand, we argue that annotations produced by our scheme can be useful to train
systems to automatically identifying different levels of genericity; these, in
turn, can be used to construct repositories of commonsense knowledge. In fact,
given their nature, generic sentences are a powerful resource to retrieve
common sense knowledge, exploitable to boost performance in various NLP
applications, such as search, question-answering, and conversational bots.
However, it is only recently that their usefulness in this regard has been
proposed and demonstrated Bhakthavatsalam et al. (2020); Nguyen et al. (2023).
We propose that the possibility of identifying sentences not only as generic,
but also being associated with values that model their semantics in a
theoretical founded way, can help in constructing resources of this kind, that
can be valuable for NLP applications, as well as for linguistics, providing
real-language data to be used in studies on the semantics of genericity.

Our framework is designed to be language-independent. In fact, as we have
seen, the aspect of genericity is a very high-level semantic phenomenon, found
in all languages yet in none explicitly marked.

In this work, we focus exclusively on NPs’ genericity, leaving aside that of
predicates, which emerges in propositions that do not express specific
episodes or isolated facts, but instead report a kind of a general property or
a generalization over events, as in John drives to work every day Krifka et
al. (1995). By that, we do not mean to argue that the two are unrelated:
indeed, very often the meaning of the predicate determines the interpretation
of the noun to which it refers as generic or non-generic. However, we are
interested in first proposing an annotation scheme that is ideally as
informative as possible for nouns, since the sentences that refer to
categories are the ones useful for the retrieval of commonsense knowledge.

### Contributions

  * •

We propose a novel annotation framework for the annotation of NPs’ genericity
that: 1. is based on simple and intuitive evaluation mechanism and labels, so
that it is suitable for crowd-sourced tasks; and 2. is grounded in the
theoretical and cognitive literature on genericity (§3).

  * •

We conduct a pilot study using this framework (§4), through which we produce a
first small annotated dataset (324 sentences)222This dataset is not yet
publicly released because it is currently being expanded and extended to other
languages, with the objective to use it in a shared task in an upcoming
evaluation campaigns. In due course, the dataset will be available at
https://osf.io/8w6u9/?view_only=9e9365d5bb8f4dba83b4081112e703ce.

  * •

We evaluate our continuous annotations comparing them with pre-existing binary
annotations on the same dataset (§5).

## 2\. Related work

One of the first frameworks that explicitly aims at modelling semantic aspects
of genericity was developed under the ACE-2 program (Mitchell et al., 2003;
Doddington et al., 2004). This framework labels NPs as SPECIFIC when they
refer to a precise member of the category and GENERIC when they refer to any
member of the category. The ACE-2005 Multilingual Training Corpus Walker et
al. (2006) extends the annotation guidelines, adding two additional classes:
negatively quantified entries (NEG) and underspecified entries (USP), where
the referent is ambiguous between GENERIC and SPECIFIC. The Situation Entities
(SitEnt) framework (Friedrich and Palmer, 2014; Friedrich et al., 2015;
Friedrich et al., 2016) improves the previous approach in two ways: first, it
annotates both NPs and entire clauses for genericity. Second, with regard to
the NPs annotation, the SitEnt guidelines, based on semantic theory, improve
and clarify the ACE guidelines, in which the notions of genericity and
specificity are conflated: the label GENERIC is applied when the subject of
the clause refers to a kind or to an arbitrary members of a kind, while the
label NON-GENERIC is applied when it refers to a particular individual.

The problematic nature of multi-class approaches is first addressed by
Govindarajan et al. (2019), who proposed a semantic framework in which
expressions of generalization are captured in a continuous multi-label system.
Their argument protocol involves the assignment of the labels PARTICULAR
(instantiated individuals), KIND (kinds of individuals), and ABSTRACT
(intangible) to each noun in argument function, together with the indication
of the annotator’s confidence about the different referential properties.

We agree with the problematization of previous frameworks advanced in
Govindarajan et al. (2019) and with the idea of a continuous type of
annotation. However, we argue that also their framework falls short in
modelling in an accurate way the gradual nature of the semantics of
genericity, which, in our view, is a key aspect of it. In fact, the indication
of annotators’ confidence is not a direct measure of continuity between
labels: their framework still assign labels to word occurrences, being
ineffective in differentiate among different types of generics.

In our framework, we bring the two labels "kind" and "particular" back onto a
single semantic axis, treating them as the two poles of a continuum, rather
than considering them separate classes or different properties. We
operationalize this dimension as inclusiveness. On the other hand, we take
into account the semantic dimension of abstractness, which is also continuous.
We argue, drawing on the descriptive and theoretical literature, that using
both dimensions can help us model NPs genericity in a more informative way, as
we will explain in the next section.

## 3\. Annotation Framework

Through our annotation framework, we propose to model genericity through (i)
two different semantic dimensions; (ii) continuous evaluations.

The use of the two dimensions that we label inclusiveness and abstractness
draws on the theoretical literature on genericity. As Chatzigoga (2019) points
out, we can divide theoretical accounts of genericity into two broad
categories: those that treat generics as quantificational, for which generics
quantify over members of the kind; and those that do not, for which generics
are seen to predicate a property directly of the kind itself. These two views
are the same that Tessler and Goodman (2019) call statistical and conceptual
views of generic language. Through inclusiveness, we aim to grasp the
quantificational, or statistical, aspect of genericity; through abstractness,
we aim to grasp the non-quantificational, or conceptual, aspect of genericity.

As for the use of continuous evaluations, we argue that it is the best way to
model a range of phenomena that are impossible to capture through a discrete
label system, such as those mentioned in §1. The most commonly used method for
capturing fine semantic properties is certainly the use of Likert scales,
typically with 5, 7 or even 9 points Brysbaert et al. (2014); Gregori et al.
(2020); Montefinese et al. (2014); however, continuous scales have many
advantages over those, as remarked in many studies Champney (1941); Svensson
(2000); Belz and Kow (2011); Ethayarajh and Jurafsky (2022). From a
psychological perspective, continuous scales are generally preferred by
raters, as they do not need to discretise their choices. Another big advantage
of continuous scales against Likert scales is that they avoid ordinal-cardinal
conflation and potentially biased estimation Ethayarajh and Jurafsky (2022).
Continuous scales in Natural Language Processing (NLP) have been mainly used
to evaluate the quality of the output of Natural Language Generation (NLG)
systems Gatt et al. (2009); Bojar et al. (2017). Beltz and Kow Belz and Kow
(2011) directly compare the two scaling methodology (i.e. graduated scales vs.
continuous scales) in the task of qualitative assessment of Natural Language
Generation (NLG) systems, concluding that the two metholodogies are
interchangable and stable in the resulting annotation. We argue that
continuous scales are also suitable for the annotation of high-level semantic
properties such as our dimensions. With our pilot study, we implicitly test
also this claim: obtaining good reliability values confirms the suitability of
the method.

We present the annotators with continuous sliders333From which values ranging
from 0 to 1 will then be extracted., on which to evaluate the two dimensions,
each in a separate task. As shown in Figure 1, our protocol proposes to the
annotators groups of sentences (from a minimum of 4 to a maximum of 8), all
containing the same noun, to be evaluated using the same scale. We chose to
display groups of sentences containing the same target noun in different
contexts because genericity, and consequently the two continuous dimensions in
which we unpack it, are relational properties. A word can be considered more
generic or less generic compared to a related word, or to the same word used
in a different context. To carry on the annotation in the correct way,
annotators need to mentally put into relation the referent of the NP in each
context to all the other existing referents denoted by the same word.
Presenting sentences in groups of this type is a way to simplify this process
for annotators, given the complex nature of the variables.

The complete instructions provided to the annotators are available in the OSF
repository of this
study444https://osf.io/8w6u9/?view_only=9e9365d5bb8f4dba83b4081112e703ce.

Figure 1: Examples of annotation interface for inclusiveness (top) and
abstractness (bottom).

#### Inclusiveness

By inclusiveness, we refer to the quantificational aspect of the semantics of
an NP, that is, how many members of the category the NP refers to. The number
of members of the category who possess the property is called prevalence in
literature on genericity; we use the term "inclusiveness" to refer not only to
generic NP, but also non-generic ones. We draw inspiration from Herbelot and
Copestake (2010). They point out that many NPs are not explicitly quantified;
still, humans are able to give them quantificational interpretations in
context (e.g., Cats are mammals = All cats…; Water dripped through the ceiling
= Some water…). The authors label this phenomenon ’underquantification’, and
argue that it also applies to generic NPs.

Their annotation scheme assumes a three-fold quantificational space,
corresponding to the quantifiers some, most and all, in addition to one, for
singular and unique entities. Following Tessler and Goodman (2019), who
propose a generalization of underquantification to a continuous interval of
possible meanings,we adopt a continuous quantificational space, in which
quantifiers are used as a reference (Figure 1, top).

We argue that the use of a quantification continuum makes it fairly intuitive
to account for the inclusiveness of NPs. From our perspective, such a scheme
is felicitously applicable to nouns referring to abstract entities as well. In
fact, according to Moltmann Moltmann (2004, 2013), bare singular abstract
nouns, such as that in (5a), are kinds, in that they denotes kinds of tropes,
while ‘tropes’ are specific instances of property attribution, such as the
same noun in (5b).

  1. 5.

    1. (a)

Ordinariness is boring.

    2. (b)

John’s ordinariness is disarming.

Such a contrast is not straightforward to capture through the assignment of
labels such as generic/non-generic or kind/particular, but we propose that can
be more easily detected by the use of a quantification continuum.

Herbelot and Copestake (2010) also claim that while generics can always be
quantified, their semantics may involve more than quantification. This seems
to be especially true for direct kind predications. Herbelot and Copestake
Herbelot and Copestake (2010, 2011), following the formal accounts of
Chierchia (1998) and Krifka (2003), argue that also generics of this type do
not preclude quantification, but that in certain cases a description of the NP
both as a quantified entity and a kind is nevertheless desirable. This
position is close to that of Tessler and Goodman (2019), who propose a
semantic core based on prevalence, but argue that prevalence is not enough to
capture subtle sensitivities to context that generic language shows. This is
precisely why we incorporated another dimension into our framework, to try to
account for the multidimensional nature of genericity.

#### Abstractness

Through abstractness, we aim to capture another aspect of the semantics of an
NP, that is, to what extent the NP denotes a referent that can be experienced
through sensory modalities (Figure 1, bottom). The first reason we
incorporated this dimension is that there is a possibility that concrete
concepts’ kinds are perceived to be more abstract than concrete concepts’
instances (Pelletier, 2009;Zamparelli, 2020). This seems to apply primarily to
direct kind predications, since in such cases the predicate is not applicable
to the individual concrete objects, but only to the kind.

Moreover, there are many cases of fine-grained polysemy in which the same word
has multiple senses that are closely related but differ in abstractness.
Consider the case of church, which can refer to the building or the social
group, depending on the context; or book, which can refer to the physical
object or the content. Similarly, there are abstract nouns whose countability
status can shift depending on the context in which they occur, and is
determined by their designation or construction in a particular occurrence
Grimm (2014). These nouns (e.g., agreement (i.e. a state of concord) vs. the
recently signed agreements), afford ’elastic’ changes between their mass and
count use. We argue that they afford also changes in their degree of
abstractness Zamparelli (2020).

We claim that the aspect of abstractness is somehow conflated into a coarse-
grained classification such as the GENERIC/NON-GENERIC one. We therefore
separate inclusiveness and abstractness while restoring their continuous
nature. This approach allows us to model the semantics of genericity in a
finer-grained manner, which arguably better captures the complexity found in
human interpretations.

## 4\. Pilot Study for Framework Validation

We conducted a pilot study to validate our annotation framework from two
different points of view: (i) whether the annotators show fair agreement on
the evaluations; (ii) whether our framework, based on annotations provided by
crowd workers, subsumes the binary GENERIC/NON-GENERIC distinction annotated
by experts. This second point (which we will discuss in §5) is necessary to
ensure that the evaluations collected actually reflect the semantic aspect of
genericity, on which the previous annotations are grounded; at the same time,
it aims to show that our scheme adds information with respect to them.

#### Dataset

We used a sample of 324 sentences extracted from the SitEnt dataset (Friedrich
and Palmer, 2014; Friedrich et al., 2015; Friedrich et al., 2016). On this
sample, we annotated the mainReferent (as defined by SitEnt) of each sentence,
that was already labeled as GENERIC or NON-GENERIC. The unique nouns in the
dataset are 60, and each occurs in a minimum of 4 to a maximum of 8 different
sentences, which are presented to the annotators in groups (Figure 1). We
tried to keep the sample as balanced as possible between GENERIC and NON-
GENERIC NPs, with 165 NON-GENERIC and 159 GENERIC target nouns; moreover, each
unique noun occurs at least once as GENERIC and once as NON-GENERIC.

Furthermore, we kept the ratio between concrete and abstract target nouns
constant with respect to that of the whole SitEnt dataset (70-30% ca.). We
associated with each mainReferent the concreteness value for the corresponding
lemma retrieved from Brysbaert et al. (2014) and considered as concrete those
nouns associated with scores greater than 3 and as abstract those nouns
associated with scores lower than or equal to 3 (Brysbaert et al. (2014)’s
scale ranges from 1 to 5).

#### Annotators

A total of 480 crowd workers, native speakers of English, were recruited as
annotators through the Prolific platform; 240 annotated inclusiveness, 240
annotated abstractness. Sentence groups were presented in batches of 6, 8 or
10, with each target noun annotated by 30 annotators. The SitEnt annotators
were highly trained and were provided with the entire document containing the
sentence to be annotated. In contrast, we used untrained annotators who were
not provided with the document but only with isolated sentences, as in
Govindarajan et al. (2019) annotation.

#### k-Rater Reliability

We evaluated the reliability of our data as k-rater reliability (kRR), which
is a multi-rater generalization of inter-rater reliability (IRR), following
the proposal of Wong and Paritosh (2022). They argue that when aggregate
ratings are used as final values, as in our case, k-rater reliability should
be used as the correct data reliability. They point out that IRR reports the
reliability of the labeling process, while kRR quantifies the reliability of
the aggregated data we consume; thus, this seems to be the correct way to
account for the reliability of the final data itself.

To compute the kRR we used the Intraclass Correlation Coefficient (ICC), a
reliability coefficient for continuous or ordinal rating scales, commonly used
in behavioral measurement and psychometrics. ICC gives researchers granular
control over assumptions about raters. For example, it is possible to define
whether each item is annotated by the same group of raters, or different
groups of raters (interchangeability). We used the latter formulation, which
allows us to generalize our reliability results to all raters who possess the
same characteristics as the raters selected in the reliability study.

The ICC for k-rater averages is denoted as ICC(k), where k stands for the
total of the raters (k = 30), following McGraw and Wong (1996) notation. For
transparency, in Table 1 we report both ICC(k) and ICC(1), that is the ICC for
the reliability of individual ratings (IRR), emphasising that the first one is
the one that account for the reliability of the aggregated data.

| ICC(k) | ICC(1)
---|---|---
inclusiveness ratings | 0.97 | 0.52
abstractness ratings | 0.94 | 0.34

Table 1: ICC(k) and ICC(1) for inclusiveness and abstractness ratings

## 5\. Analysis

To demonstrate that our continuous scheme subsumes the standard distinction
GENERIC vs. NON-GENERIC we compared the aggregated data derived from our
annotation with the SitEnt gold annotations. We will refer to the average of
inclusiveness ratings as INC and that of abstractness ratings as ABS.

### 5.1. Quantitative comparison

Predictor(s) | SitEnt label | P | R | F
---|---|---|---|---
| INC
---
(accuracy = 0.78)
NON-GENERIC | 0.77 | 0.82 | 0.79
| GENERIC | 0.82 | 0.74 | 0.77
| ABS
---
(accuracy = 0.68)
NON-GENERIC | 0.65 | 0.76 | 0.70
| GENERIC | 0.73 | 0.60 | 0.65
| INC + ABS
---
(accuracy = 0.80)
NON-GENERIC | 0.78 | 0.83 | 0.80
| GENERIC | 0.83 | 0.76 | 0.79

Table 2: Prediction of SitEnt labels GENERIC/NON-GENERIC using our continuous
annotations in logistic regression models. Figure 2: Distribution of INC and
ABS by SitEnt labels GENERIC/NON-GENERIC.

sentence | INC | ABS | SitEnt label
---|---|---|---
1\. Zebras evolved among the Old World horses within the last 4 million years. | 0.97 | 0.88 | GENERIC
2\. He stepped away as the car backed up. | 0.00 | 0.08 | NON-GENERIC
3\. Asian countries have tended to give priority to economic, social and cultural rights, but have often failed to provide civil and political rights. | 0.59 | 0.66 | GENERIC
4\. Many countries abolished the monarchy in the 20th century and became republics. | 0.57 | 0.68 | NON-GENERIC
5\. When the large fish of the Colossoma genus entered the aquarium trade in the U.S. and other countries, they were erroneously labeled pacu. | 0.33 | 0.75 | GENERIC
6\. When a cat is dropped, it always lands on its feet. | 0.92 | 0.53 | GENERIC

Table 3: Some sentences from our dataset; target nouns are underlined.

Preliminarily, we performed a Wilcoxon rank-sum test to assess whether the
difference in INC and ABS between the GENERIC and NON-GENERIC group was
statistically significant. The result is highly significant for both INC (p =
1.81e-31) and ABS (p = 1.14e-15).

Then, we fit three logistic regression models to predict the SitEnt label of
each noun on the basis of our annotations: one using only INC as predictor,
one using only ABS and one using both. To select the hyperparameters for these
classifiers we used grid search over different solvers (solver in {’newton-
cg’, ’lbfgs’, ’liblinear’, ’sag’, ’saga’}) and the regularization parameter C
(C in {0.001, 0.01, 0.1, 1, 10}) in a 5-fold stratified crossvalidation (CV)
nested within a 10-fold CV555To fit the regression models we relied on the
scikit-learn Library: https://scikit-learn.org/stable/index.. The standard
deviation for each metric across the 10 folds is always < 0.15. This means
that the performance metrics are stable across folds and that the models are
not overly sensitive to the specific subsets of data used for training. The
metrics for each classifier (accuracy; precision, recall and F1-score for each
class), computed from this 10-fold CV, are reported in Table 2.

The results show good performances of the models, which means that our
annotations provided by crowd workers are good predictors for binary
classification between the SitEnt labels GENERIC and NON-GENERIC, provided by
experts.

Interestingly, all three models perform well, and the best-performing one is
that using both INC and ABS as predictors (although the difference is only a
few points compared to the model using only INC). This suggests that both
semantic dimensions are good predictors for the aspect of genericity, and that
the information conveyed by each one adds predictive power to the other. These
results also confirm that the use of untrained annotators, who rely primarily
on their intuition, and isolated sentences rather than whole documents, still
allows us to capture information about the phenomenon of genericity.

The models perform well enough to confirm that our continuous annotation
tracks the binary one. However, the performances are still not excellent,
rising the possibility that a binary classification does not reflect the
totality of information captured by our continuous annotation. In fact, the
cases of mis-classification GENERIC/NON-GENERIC are probably due to the
difference between type of annotation (binary vs. continuous): binary labels
force evaluators to one or the other label, and in some cases this choice is
likely to be arbitrary (as we will show in the next section and is reported in
Table 3), while continuous evaluations allow for nuances.

### 5.2. Qualitative comparison

The distribution of INC and ABS in comparison with that of GENERIC/NON-GENERIC
labels is shown by the histograms in Figure 2.

The distribution of INC (left) reflects that of the SitEnt labels: nouns
labeled as NON-GENERIC cluster on low values of inclusiveness (with a peak on
the minimum value, which on our scale corresponds to ’a particular X’), nouns
labeled as GENERIC on high values of inclusiveness. This is in agreement with
our expectations: in fact, nouns annotated as GENERIC (referring to a kind or
an arbitrary member of a kind, according to the SitEnt guidelines) will tend
to be perceived as more inclusive, i.e., as referring to more elements of the
category denoted by the word, than those annotated as NON-GENERIC (referring
to a particular individual, according to the guidelines). However, the central
part of the plot shows a large overlap area, where intermediate values of
inclusiveness match both nouns labeled as GENERIC and as NON-GENERIC.

As for ABS (Figure 2, right), nouns labeled as GENERIC tend to cluster on high
ratings, while those labeled as NON-GENERIC are fairly evenly distributed
along the entire scale. Thus, in this case, the distribution of ratings
mirrors that of the binary labels with regard to the GENERIC label, as we
expected: nouns referring to a kind tend to be perceived as more abstract. The
mixed situation exhibited by nouns labeled as NON-GENERIC, on the other hand,
will be at least partly accounted for by the inherent semantics of the noun in
question: that is, inherently very abstract nouns, such as idea, will tend to
always receive high abstractness ratings. We will return to this aspect in
§5.3.

The existence of broad areas of overlap between GENERIC and NON-GENERIC nouns
in the distribution of INC and ABS confirms that a discrete classification is
unable to capture the totality of information conveyed by our continuous
annotation. The binary distinction is in full agreement with the continuous
annotations as far as they concern the cases of prototypical generic and non-
generic nouns, which will be indeed characterized by very high values for both
INC and ABS (see sentence (1) in Table 3) and very low values for both INC and
ABS (sentence (2) in Table 3), respectively.

Figure 3: Distribution of INC (left) and ABS (right) for concrete (top) and
abstract (bottom) nouns.

However, natural language is rarely so clear-cut. Our analysis shows that
there are also non-prototypical cases that are difficult to classify by binary
annotation and that could benefit from a representation through our semantic
dimensions. Consider cases (3) and (4) in Table 3: in both sentences, the word
countries does not blatantly refer either to a kind or to a specific
individual. This is reflected by intermediate values of INC and ABS, very
close for the two cases. However, in SitEnt the first one is classified as
GENERIC and the other as NON-GENERIC, which confirms the difficulty of
assigning some cases to closed classes. The last two examples in Table 3 show
that not all generics are the same, and that our annotation can usefully
capture fine-grained differences between them. Sentence (5) shows a case of
taxonomic reference, where the word fish does not refers to the category of
all fish, but to a subkind; it is also a case of direct kind predication.
These features are well represented by the ratings: the word is low in
inclusiveness because refers to a relatively small number of fish, but high in
abstractness because it still refers to a kind. In sentence (6), on the
contrary, the word cat is high in inclusiveness and low in abstractness. This
sentence is a characterizing generic, with the subject in its indefinite form:
the predication is applicable to a vast majority of individuals of the kind,
but the noun, rather than refer directly to the kind, refers to an arbitrary
member of the kind, which is perceived as less abstract.

### 5.3. Inherent semantics of words

The last point we consider in our analysis is that of the inherent semantics
of words evaluated for their in-context genericity, particularly with respect
to their concreteness/abstractness. This aspect is rarely addressed
explicitly, either by the theoretical literature on genericity or by works on
its annotation, probably because the distinction between generic and non-
generic meaning is more difficult to draw for nouns referring to abstract
entities.

Govindarajan et al. (2019) attempt to address this problem proposing the
disentanglement of the following dimensions: KIND, PARTICULAR and ABSTRACT
(non-mutually exclusive labels). However, their analysis shows that there is a
tendency for abstract-referring nouns to be neither kind-referring nor
particular-referring. Our claim is that the kind/particular (or generic/non-
generic) meaning is well represented by the dimension of inclusiveness for
both concrete nouns, as we saw in the previous paragraph, and abstract nouns,
as can be seen from the difference in INC in a contrast such as:

  1. 6.

    1. (a)

My mind took this as a challenge, something I had to prove wrong. [INC: 0.01;
ABS: 0.64]

    2. (b)

Substance dualists argue that the mind is an independently existing substance.
[INC: 0.87; ABS: 0.90]

We can rely on the dimension of inclusiveness for these cases as well, without
having to take the kind/particular meaning out of the equation for abstract
entities. This can also be inferred from the plots shown in Figure 3,
analogous to those in Figure 2, but in which the target nouns have been split
into concrete (top) and abstract (bottom)666We recall that we considered as
concrete nouns associated with scores $>$ 3 and as abstract nouns associated
with scores $\leq$ 3 in Brysbaert et al. (2014). The distribution of INC
(plots on the left) is similar between concrete and abstract nouns: in both
cases there is a bimodal trend with respect to binary labels, with the
difference that abstracts do not show the same peak as concretes on the
minimum value.

As for ABS, we claim that it is determined both by context and by the inherent
semantics of the word. The influence of context is particularly evident in the
case of concrete entities, as can be seen from the difference in ABS in a
contrast such as:

  1. 7.

    1. (a)

The cat sadly shook its head and meowed. [INC: 0.01; ABS: 0.06]

    2. (b)

The domestic cat was first classified as Felis catus by Carolus Linnaeus.
[INC: 0.85; ABS: 0.89]

ABS values are also influenced by the inherent semantics of the word, which is
particularly evident for nouns referring to abstract entities. In both (4a)
and (4b), for example, ABS values are above the midpoint of the scale: this is
because the entity in question is inherently abstract (Brysbaert et al., 2014
score: 2.5). However, interestingly, there is still a difference in ABS
between the two cases, which confirms that there is an influence of context in
these cases as well.

The influence of the word’s inherent semantics thus helps to explain the ABS
distribution of NON-GENERIC nouns in Figure 2. This can be better visualized
in the plots on the right in Figure 3. For concrete nouns (top), the ABS
distribution is bimodal between GENERIC and NON-GENERIC, similarly to the INC
one. For abstract nouns (bottom), the distribution is skewed toward high
values on the scale: in-context abstractness of inherently abstract nouns does
not fall below a certain threshold.

## 6\. Discussion and Conclusions

In this paper, we introduced a novel annotation framework aimed at capturing
the nuances of noun phrases’ (NPs) genericity in a fine-grained manner. We
discussed our dual objective in proposing this framework: firstly, to examine
if naive language users’ annotations can reveal differences in genericity that
align with phenomena observed in theoretical literature by experts; secondly,
we argue that the annotations generated through our framework can serve as
valuable training data for systems to automatically identify different levels
of genericity, which, in turn, can be employed to construct repositories of
commonsense knowledge. To validate our annotation scheme, we compared
continuous annotations collected in crowdsourcing tasks with existing binary
annotations on the same dataset, showing that our continuous annotations
reliably capture fine-grained nuances of genericity.

In conclusion, our work holds potential for linguistics, in that it provides a
first dataset of natural occurring sentences annotated according fine-grained
continuous values modeling NPs’ genericity and an annotation scheme designed
to build such annotated datasets, that can be used in semantic studies on
genericity. Furthermore, it is also exploitable for the creation of
commonsense knowledge repositories, useful for the enhancement of various NLP
applications.

## Acknowledgements

Funded by the European Research Council, Abstraction project (attributed to
Marianna Bolognesi. Grant agreement: ERC-2021-STG-101039777). Views and
opinions expressed are however those of the authors only and do not
necessarily reflect those of the European Union or the European Research
Council Executive Agency. Neither the European Union nor the granting
authority can be held responsible for them.

Authors Contribution: CC (conception, experimental design, data collection,
data analyses, writing); AAM (conception, revision); MMB (conception,
revision, supervision).

We thank the anonymous reviewers for their useful suggestions, and all the
members of the research group Abstraction
(https://www.abstractionproject.eu/).

## Bibliographical References

##

  * Behrens (2005) Leila Behrens. 2005.  Genericity from a cross-linguistic perspective.  _Linguistics_ , pages 275–344.
  * Belz and Kow (2011) Anja Belz and Eric Kow. 2011.  Discrete vs. continuous rating scales for language evaluation in nlp.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 230–235.
  * Bhakthavatsalam et al. (2020) Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020.  Genericskb: A knowledge base of generic statements.  _arXiv preprint arXiv:2005.00660_.
  * Bojar et al. (2017) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017.  Findings of the 2017 conference on machine translation (wmt17).  Association for Computational Linguistics.
  * Brysbaert et al. (2014) Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014.  Concreteness ratings for 40 thousand generally known english word lemmas.  _Behavior research methods_ , 46:904–911.
  * Carlson and Pelletier (1995) Gregory N Carlson and Francis Jeffry Pelletier. 1995.  _The generic book_.  University of Chicago Press.
  * Carlson (1977) Gregory Norman Carlson. 1977.  _Reference to kinds in English._ University of Massachusetts Amherst.
  * Champney (1941) Horace Champney. 1941.  The measurement of parent behavior.  _Child Development_ , pages 131–166.
  * Chatzigoga (2019) Dimitra Lazaridou Chatzigoga. 2019.  Genericity.  In _The Oxford Handbook of Experimental Semantics and Pragmatics_ , pages 156–177. Oxford University Press.
  * Chierchia (1998) Gennaro Chierchia. 1998.  Reference to kinds across language.  _Natural language semantics_ , 6(4):339–405.
  * Dahl (1995) Osten Dahl. 1995.  The marking of the episodic/generic distinction in tense-aspect systems.  In Greg N. Carlson and Francis Jeffry Pelletier, editors, _The Generic Book_. University of Chicago Press.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Ethayarajh and Jurafsky (2022) Kawin Ethayarajh and Dan Jurafsky. 2022.  The authenticity gap in human evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6056–6070.
  * Friedrich and Palmer (2014) Annemarie Friedrich and Alexis Palmer. 2014.  Situation entity annotation.  In _Proceedings of LAW VIII-The 8th Linguistic Annotation Workshop_ , pages 149–158.
  * Friedrich et al. (2016) Annemarie Friedrich, Alexis Palmer, and Manfred Pinkal. 2016.  Situation entity types: automatic classification of clause-level aspect.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1757–1768.
  * Friedrich et al. (2015) Annemarie Friedrich, Alexis Palmer, Melissa Peate Sørensen, and Manfred Pinkal. 2015.  Annotating genericity: a survey, a scheme, and a corpus.  In _Proceedings of the 9th Linguistic Annotation Workshop_ , pages 21–30.
  * Gatt et al. (2009) Albert Gatt, Anja Belz, and Eric Kow. 2009.  The tuna-reg challenge 2009: Overview and evaluation results.  Association for Computational Linguistics.
  * Govindarajan et al. (2019) Venkata Govindarajan, Benjamin Van Durme, and Aaron Steven White. 2019.  Decomposing generalization: Models of generic, habitual, and episodic statements.  _Transactions of the Association for Computational Linguistics_ , 7:501–517.
  * Gregori et al. (2020) Lorenzo Gregori, Maria Montefinese, Daniele P Radicioni, Andrea Amelio Ravelli, Rossella Varvara, et al. 2020.  Concretext@ evalita2020: The concreteness in context task.  In _EVALITA_.
  * Grimm (2014) Scott Grimm. 2014.  Individuating the abstract.  In _Proceedings of Sinn und Bedeutung_ , volume 18, pages 182–200.
  * Herbelot and Copestake (2010) Aurelie Herbelot and Ann Copestake. 2010.  Annotating underquantification.  In _Proceedings of the Fourth Linguistic Annotation Workshop_ , pages 73–81.
  * Herbelot and Copestake (2011) Aurelie Herbelot and Ann Copestake. 2011.  Formalising and specifying underquantification.  In _Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011)_.
  * Krifka (2003) Manfred Krifka. 2003.  Bare nps: Kind-referring, indefinites, both, or neither?  In _Semantics and linguistic theory_ , volume 13, pages 180–203.
  * Krifka et al. (1995) Manfred Krifka, Francis Jeffry Pelletier, Gregory Carlson, Alice ter Meulen, Gennaro Chierchia, and Godehard Link. 1995.  Genericity: An introduction.  In Greg N. Carlson and Francis Jeffry Pelletier, editors, _The Generic Book_ , pages 1–124. University of Chicago Press.
  * Leslie and Lerner (2022) Sarah-Jane Leslie and Adam Lerner. 2022.  Generic Generalizations.  In Edward N. Zalta and Uri Nodelman, editors, _The Stanford Encyclopedia of Philosophy_ , Fall 2022 edition. Metaphysics Research Lab, Stanford University.
  * McGraw and Wong (1996) Kenneth O McGraw and Seok P Wong. 1996.  Forming inferences about some intraclass correlation coefficients.  _Psychological methods_ , 1(1):30.
  * Moltmann (2004) Friederike Moltmann. 2004.  Properties and kinds of tropes: New linguistic facts and old philosophical insights.  _Mind_ , 113(449):1–41.
  * Moltmann (2013) Friederike Moltmann. 2013.  _Abstract objects and the semantics of natural language_.  Oxford University Press.
  * Montefinese et al. (2014) Maria Montefinese, Ettore Ambrosini, Beth Fairfield, and Nicola Mammarella. 2014.  The adaptation of the affective norms for english words (anew) for italian.  _Behavior research methods_ , 46:887–903.
  * Nguyen et al. (2023) Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023.  Extracting cultural commonsense knowledge at scale.  In _Proceedings of the ACM Web Conference 2023_ , pages 1907–1917.
  * Pelletier (2009) Francis Jeffry Pelletier. 2009.  _Kinds, things, and stuff: Mass terms and generics_.  Oxford University Press.
  * Svensson (2000) Elisabeth Svensson. 2000.  Comparison of the quality of assessments using continuous and discrete ordinal rating scales.  _Biometrical Journal: Journal of Mathematical Methods in Biosciences_ , 42(4):417–434.
  * Tessler and Goodman (2019) Michael Henry Tessler and Noah D Goodman. 2019.  The language of generalization.  _Psychological review_ , 126(3):395.
  * Wong and Paritosh (2022) Ka Wong and Praveen Paritosh. 2022.  k-rater reliability: The correct unit of reliability for aggregated human annotations.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 378–384.
  * Zamparelli (2020) Roberto Zamparelli. 2020.  Countability shifts and abstract nouns.  _Mass and Count in Linguistics, Philosophy, and Cognitive Science. Benjamins, Amsterdam_.

## 7\. Language Resource References

##

  * Mitchell et al. (2003) Mitchell et al. 2003.  _ACE-2 Version 1.0_.  Linguistic Data Consortium, Philadelphia, PA., 0.1, ISLRN 498-363-793-174-9.
  * Walker et al. (2006) Walker et al. 2006.  _ACE 2005 Multilingual Training Corpus_.  Linguistic Data Consortium, Philadelphia, PA., ISLRN 458-031-085-383-4.