# On Package Freshness in Linux Distributions
††thanks: Fonds de la Recherche Scientifique – FNRS

Damien Legay  Software Engineering Department
University of Mons
Mons, Belgium
<EMAIL_ADDRESS>Alexandre Decan  Software Engineering Department
University of Mons
Mons, Belgium
<EMAIL_ADDRESS>Tom Mens  Software Engineering Department
University of Mons
Mons, Belgium
<EMAIL_ADDRESS>
###### Abstract

The open-source Linux operating system is available through a wide variety of
distributions, each containing a collection of installable software packages.
It can be important to keep these packages as fresh as possible to benefit
from new features, bug fixes and security patches. However, not all
distributions place the same emphasis on package freshness. We conducted a
survey in the first half of 2020 with 170 Linux users to gauge their
perception of package freshness in the distributions they use, the value they
place on package freshness and the reasons why they do so, and the methods
they use to update packages. The results of this survey reveal that, for the
aforementioned reasons, keeping packages up to date is an important concern to
Linux users and that they install and update packages through their
distribution’s official repositories whenever possible, but often resort to
third-party repositories and package managers for proprietary software and
programming language libraries. Some distributions are perceived to be much
quicker in deploying package updates than others. These results are valuable
to assess the requirements and expectations of Linux users in terms of package
freshness.

##  I Introduction

The Linux operating system is arguably the most successful open-source project
to ever have come to fruition. Per the nature of open-source software, Linux
is available in many forms, called distributions. Each distribution is
composed of the Linux kernel and a host of software products provided in the
form of packages. The number of these packages in Linux distributions is known
to grow superlinearly [1, 2]. Packages are made available to users through a
wide variety of package managers. Some, such as pacman, dkpg or RPM, are
specific to a distribution and its derivatives whilst others, such as Flatpak
or Snappy, are intended for cross-distribution usage. The set of packages that
can be found within these package managers largely depends on the ambitions
and philosophy of the distribution. Some distributions, such as Debian, pledge
that all their components will be entirely composed of free
software111https://www.debian.org/social_contract while other distributions
make no such promise. Philosophical divergences also cause versions of
packages available in each distribution to differ, as distributions weigh
concerns of stability (the ability of a distribution to withstand changes in
its components) and package freshness (how up to date a package is compared to
its upstream releases) differently.

This presents a trade-off to the maintainers of Linux distributions. On the
one hand, adopting new versions of packages within the distribution will grant
users access to new features, bug fixes and security patches. On the other
hand, these new versions risk introducing breaking changes, new bugs, security
vulnerabilities, incompatibilities or co-installability issues wherein
packages cannot be installed without creating conflicts with some other
packages [3, 4]. Specifically for Debian, Claes et al. estimated that the
number of packages being incompatible with at least one other package
oscillates between 15% and 25% over time [5]. Distribution maintainers thus
have to go through the very time-consuming process of assessing new versions
of packages for these risks.

Reliance on semantic versioning222https://semver.org could mitigate the cost
of assessing for breaking changes but it is known that different programming
language ecosystems comply to semantic versioning to different degrees [6].
This implies that, depending on the language a package is written in, even
patch or minor releases are likely to contain breaking changes. Specifically
for Maven, Raemakers et al. [7] observed that about one third of all updates,
including minor releases and patches, introduce API breaking changes.
Similarly, the task of assessing updates for incompatibilities depends on the
accuracy of the package metadata, but that metadata is often invalidated
through package evolution [8]. These factors help explain the choice of some
distribution maintainers to emphasise stability and security over package
freshness. Yet, package freshness is important to the Linux community, as
evidenced by the existence of package freshness monitoring services such as
Repology and DistroWatch.

Our goal is to measure, understand and compare package freshness in Linux
distributions and evaluate to which extent the perception of users matches
reality. To achieve this goal, we will conduct a mixed study, consisting of a
qualitative component (a survey of Linux users) and a quantitative component
(empirical analyses on the freshness of packages in Linux distributions). This
paper constitutes a first step towards this goal, reporting upon the results
of the survey.

##  II Related Work

Little is known about package freshness in Linux distributions in general.
Shawcroft [9] compared the package freshness of a limited set of 37 packages
in 8 Linux distributions by looking the number and proportion of obsolete
packages, measuring the time between upstream release of a package version and
downstream deployment into a distribution, as well as quantifying the number
of upstream versions that are ahead of deployed versions. We will extend this
work to a larger corpus of packages, using more recent data. Gonzalez-Barahona
et al. [10] observed that one out of eight packages (12%) was not updated at
all within a nine-year timespan, from Debian Stable 2.0 (released on
1998-07-24) to 4.0 (released on 2007-04-08). Nguyen and Holt [11] studied the
lifecycle of Debian packages. They compared the age of packages in Debian
distributions Unstable, Testing and Stable, defining package age as the time
delta between its introduction into the distribution and its removal from
Debian or its replacement (update) by a newer version of the same package.

The freshness of distributed packages has been formalised by the notion of
technical lag (expressed both in terms of time and number of versions) by
Gonzalez-Barahona et al. [12, 13]. Zerouali et al. [14] used technical lag to
explore how outdated packages are in Debian-based Docker containers and the
impact that such outdated packages have on the presence of security
vulnerabilities and bugs. Decan et al. [15] used it to assess the reluctance
of package maintainers to update package dependencies in order to avoid
putative backward-incompatible changes in programming language ecosystems.

##  III Methodology

This paper reports on a survey of Linux users, conducted in early 2020, aiming
to examine their perception of package freshness. We specifically explore the
value Linux users place on package freshness, their motivations to upgrade
packages to newer versions, the way they use to do so and how fresh they
perceive the packages in their most used distribution to be. In order to
obtain a sampling of the views of the open-source community at large, we
distributed the survey to attendants of the Free and Open source Software
Developers’ European Meeting (FOSDEM 2020) and Community Health Analytics Open
Source Software conference (CHAOSSCon Europe 2020), in both paper and
electronic versions. We received 68 responses, 52 from CHAOSSCon and FOSDEM, 9
from convenience sampling and 7 from computer science students at our
university. 37 answers came in paper form and 31 were submitted online. We
also posted the survey on Twitter and Linux-related fora and subreddits,
obtaining an additional 102 responses, for a total of 170. The fora were:
linux.org, forums.fedoraforum.org, forums.debian.net, forums.linuxmint.org and
the subreddits /r/centos, /r/fedora, /r/redhat, /r/linuxmint and /r/opensuse.
We did not receive authorisation to post the survey on other Linux-related
fora and subreddits. The survey form and anonymised answers can be found at
https://doi.org/10.5281/zenodo.3908332.

Since the answers to the survey questions can depend on the distribution(s)
used, we asked respondents which Linux distribution(s) they frequently make
personal use of. They could rank up to three distributions, in order of
frequency of use, chosen from a pre-established list of 16 popular Linux
distributions. They also had the option to specify other distributions.

Table I reports the total number of answers obtained for each distribution, as
well as the number of times that distribution was ranked first, second or
third. The table also reports the aggregated responses for each family of
Linux distributions. Distributions for which there were fewer than five
answers have been gathered under the label “other distributions”. These are
Gentoo, SUSE Entreprise Edition, Parabola, FerenOS, Android, Clear Linux,
Knoppix, Alpine Linux, NixOS and Raspbian. These results indicate that 88%
(149) of the respondents make use of at least two distributions, and 62% (106)
of at least three, showing that most of them have experience with several
distributions, either from using them concurrently or migrating from one to
another. Often, people who use more than one distribution use one that favours
stability (e.g. CentOS) and one that favours freshness (e.g. Fedora).

TABLE I: Usage per (family of) Linux distribution(s). Distribution | First | Second | Third | Total
---|---|---|---|---
Ubuntu (family) | 47 | 46 | 43 | 117
Ubuntu LTS | 30 | 25 | 11 | 66
Ubuntu | 17 | 16 | 18 | 51
Debian (family) | 30 | 37 | 26 | 93
Stable | 19 | 24 | 22 | 65
Testing | 10 | 13 | 4 | 27
Unstable | 1 | 0 | 0 | 1
Red Hat (family) | 33 | 33 | 25 | 91
Fedora | 24 | 9 | 8 | 41
CentOS | 8 | 16 | 9 | 33
Entreprise Edition | 1 | 8 | 8 | 17
Arch Linux | 29 | 8 | 8 | 45
OpenSUSE (family) | 21 | 13 | 5 | 39
Tumbleweed | 17 | 4 | 2 | 23
LEAP | 4 | 9 | 3 | 16
Linux Mint | 5 | 10 | 7 | 22
Slackware | 2 | 2 | 1 | 5
Other distributions | 3 | 6 | 5 | 14
Total | 170 | 149 | 106 |

##  IV Findings

###  IV-A Which distributions are perceived to be more up-to-date?

To gauge the user perception of package freshness in Linux distributions, we
asked respondents how long it took, according to them, for the latest upstream
version to be made available in the official repositories of their most-used
distribution (answered first in the previous question). We gave them six
exclusive options: “a few days, at most”, “a few weeks”, “a few months”, “a
few years”, “never (not available)” and “I don’t know”. We asked about six
categories of packages:
_OSS: open-source end-user software_ (e.g. Firefox or GIMP);
_PS: proprietary end-user software_ (e.g. Adobe Reader, Spotify or Skype);
_DT: development tools_ (e.g. git, Emacs or Eclipse);
_STL: system tools and libraries_ (e.g. openSSL, sudo or zsh);
_PLL: programming language libraries_ (e.g. NumPy for Python, Lodash for npm)
_PLR: programming language runtimes_ (e.g. Python, node.js or Java).

Fig. 1 presents the median answer for each category. Only distributions which
are used primarily by five or more respondents are shown.

---
Figure 1: Package freshness perception.

Being based on a rolling release policy, Arch Linux and OpenSUSE Tumbleweed
strive to distribute the latest stable releases of all packages included in
the distribution. Fig. 1 reveals that the respondents’ perception aligns with
this reality, since most of them agree that upstream versions are made
available very quickly, within days. Only proprietary end-user software (PS)
packages are perceived by some to be updated slower than a matter of weeks,
when at all available, in accordance with the fact that some distributions do
not directly support proprietary software.

Fedora users believe that they will dispose of fresh versions within weeks.
Most Fedora users answered that proprietary software just was not available in
Fedora’s official repositories. Ubuntu, Linux Mint and Debian Testing users
usually think it takes weeks to months for upstream versions to be released
within the official repositories. Although the median answer for proprietary
software in Debian Testing is that it takes years, this is due to the fact
that a significant portion of respondents answered proprietary software was
not available in Debian Testing. At the end of the spectrum, CentOS and Debian
Stable users tend to expect to wait months for fresh versions to be made
available in their distribution’s official repositories, regardless of package
type. 7 out of the 170 respondents expressed that they did not know when
updates are made available to the distribution’s official repositories. A
further 50 respondents expressed ignorance for only some categories,
principally regarding proprietary software.

###  IV-B To what extent do users value package freshness?

We enquired, for each package category, what importance users impart to
keeping packages up to date with upstream releases. To do so, we relied on a
4-value Likert scale to denote relative importance: unimportant, slightly
important, moderately important and very important. Fig. 2 reports on the
results for each package category.

For all categories, respondents consider it important to update packages: 75%
to 80% of them answered it was moderately to very important to remain up to
date. A notable exception is the proprietary end-user software category (PS):
in this instance alone, a majority (52%) considers the importance of package
freshness to be slight or null. We do not see a clear practical reason why
users would consider updating proprietary packages less important than other
packages.

---
Figure 2: Importance imparted by respondents to staying up-to-date

Users of distributions that are perceived to be slower in deploying packages,
such as CentOS and Debian Stable, were less likely to value maintaining a high
level of package freshness. Conversely, users of distributions that are
perceived to have fresher packages, such as Arch Linux, OpenSUSE Tumbleweed
and Fedora were more likely to consider package freshness important. Indeed,
across all categories, a much greater proportion of respondents answered that
updating packages was moderately to very important for Arch (80%), Tumbleweed
(80%) and Fedora (84%), than for CentOS (50%) and Debian Stable (54%).

###  IV-C What are the main reasons for updating packages?

Benefits to updating packages include access to new features, bug fixes and
security patches. In order to verify whether those benefits actually motivate
users to update, respondents were asked what their main reasons were to update
packages, out of five options: to benefit from security patches (selected by
90% of the respondents), from bug fixes (80%), from new features (66%), to
sate their desire to remain up to date (35%) or to retain compatible with
other packages (27%). They could select as many options as they wanted. An
additional open option was available, but not used by any respondent. The
motivation of obtaining new features is less prevalent in users of
distributions that are perceived as less fresh, such as Debian Stable and
CentOS. Users of Debian are less likely to cite bugs as reasons to update,
likely owing to the Debian process of package integration leading to stable
distributions, particularly Debian Stable.

###  IV-D Which mechanisms are used to keep packages up-to-date?

Several mechanisms can be used to update packages: using the _official package
manager of the distribution and its official repository ( off)_, using the
_official package manager of the distribution with community repositories (
com)_, using _third-party package managers such as Flatpak (3rd)_, _installing
manually from binaries ( bin)_ or _installing manually from source files (
src)_.

We asked respondents which of these mechanisms they used to update packages in
their most-used distribution, for each of the considered package categories.
They could select as many answers as they wanted. Fig. 3 shows a heatmap of
the frequency at which respondents reported which of these mechanisms is used,
for each package category.

---
Figure 3: Frequency of updating mechanism usage.

For most package categories, the official repositories (off) dominate largely
(used by 79% to 94% of respondents), followed by community repositories (used
by 19% to 41%). Proprietary software stands in contrast to other categories,
being installed more than a third of the time from binaries (bin). This is
expected, as some distributions (e.g. Debian) are reluctant to include
proprietary software within their official repositories. We also see that,
despite their recency, third-party managers (3rd) such as Flatpak or Snappy
are regularly used to install end-user open source software (21%), development
tools (22%) or programming language libraries (29%). We also observe that
programming language libraries are less often installed through official
repositories, and are installed nearly one third of the times through specific
third-party package managers. This should be no surprise given that most
libraries for these languages are (sometimes exclusively) available through
dedicated package managers (e.g. pip for Python, npm for Javascript). In the
case of packages related to development tools, we believe this is likely due
to the assortment of tools available through Flatpak and Snappy, including
popular IDEs, text editors and graphical user interfaces for git. These tools
are not always available in official or community repositories. For instance,
Intellij IDEA is not available in the official repositories of Fedora, but can
be installed through Flatpak and Snappy.

##  V Discussion

We reported the results of a survey of 170 Linux users about package
freshness. We found that users perceive distributions such as Arch Linux,
OpenSUSE Tumbleweed and Fedora as being much more likely to have fresh
packages than distributions such as Debian Stable and CentOS. Verifying these
perceptions will require a quantitative empirical comparison of freshness in
Linux distributions.

As a preliminary step towards such an empirical study, we gathered the package
versions available in a recent snapshot of the five distributions respondents
cited the most. We took the latest snapshot available prior 2019-11-01. This
date was selected to minimise the gaps between distribution release dates.
Ubuntu 19.10 was chosen over LTS versions of the distribution for that reason.
The distributions considered are listed in Table II. We identified 529 common
packages for these distributions. We performed pairwise comparisons on the
freshness of the versions of these packages present in those distributions.
The results are found on Fig. 4 with each cell reporting the proportion of
packages that are at least as fresh in the source distribution as in the
target distribution. For instance, 99% of packages in Arch Linux are at least
as fresh as those in CentOS, whereas only 28% of packages in CentOS are at
least as fresh as those in Arch Linux. This means that 72% of the packages in
CentOS are outdated with respect to those available in Arch Linux.

TABLE II: Releases of the considered Linux distributions Distribution | Release | Date
---|---|---
Arch Linux | rolling | 2019-10-31
CentOS | 8.0 | 2019-09-24
Debian Stable | 10 | 2019-07-06
Fedora | 31 | 2019-10-29
Ubuntu | 19.10 | 2019-10-17
---
Figure 4: Proportion of packages in a source distribution that are at least as
fresh as in a target distribution

We see that almost all packages in Arch Linux are at least as fresh as the
packages found in the other distributions, and only 28% (CentOS) to 72%
(Fedora) of packages in other distributions are as fresh as Arch Linux
packages. At the other extreme, the vast majority of packages (94% or more) in
all considered distributions are at least as fresh as those found in CentOS
and only 28% to 53% of CentOS packages are at least as fresh as those found in
other distributions. Debian Stable is in a similar situation as CentOS, albeit
slightly less marked. Fedora and Ubuntu lie in the middle, with 72% and 65%
(resp.) of their packages as fresh as those found in Arch.

These results suggest the following ranking of distributions in decreasing
order of freshness: Arch Linux, Fedora, Ubuntu, Debian Stable and finally
CentOS. This roughly corresponds to respondent perception in Fig. 1.
Nevertheless, these perceptions are imprecise, as evidenced by the fact that
Ubuntu LTS users considered some categories of packages to be available within
their distribution’s repositories sooner than non-LTS Ubuntu users, which is
contrary to expectations. Additionally, 57 respondents (33%) were not
confident to answer for at least one package category, with 7 (4%) of them
answering they did not know for all categories.

This preliminary empirical analysis of package freshness in five Linux
distributions hints to the fact that distributions lie on a continuum with
regards to the trade-off between package freshness and system stability. In a
follow-up study, we will seek to empirically quantify the difference in
package freshness between distributions by relying on the technical lag
measurement framework [13]. We will compare the versions of packages available
in different distributions in terms of time lag (i.e. the time since a more
recent upstream version of the package has been available) and version lag
(i.e. the number of more recent versions available). This will allow us to
place distributions on that continuum, helping users choose a distribution
that best matches their expectations in terms of freshness and stability. We
will seek to contrast the package freshness measured within distributions with
the perceptions of users reported in this paper, thereby gauging to what
degree user perception matches reality.

We will also examine the relationship between package freshness, stability and
security in distributions. Comparing distributions in terms of these
characteristics could help users to choose a distribution that matches their
requirements and expectations. It will also allow package maintainers to know
whether their packages are likely to be up to date in certain distributions,
and potentially adopt practices that allow distribution maintainers to assess
the stability, compatibility and security of their packages more quickly, to
allow faster deployment of updates. Additionally, we will conduct a follow-up
survey to examine to what extent package freshness, stability and security
motivate users to migrate from one distribution to another.

##  VI Conclusion

The Linux ecosystem depends on a set of packages. These packages are made
available through package managers, either official ones used by specific
distributions, or third-party ones, as well as directly through binaries and
source files. Package versions available in official distribution repositories
do not always match the latest versions released by the package’s authors, out
of a need to balance package freshness with system stability and security.

This paper is a first step towards a mixed study to understand, measure and
compare package freshness in Linux distributions. We reported on the results
of a survey aimed at habitual Linux users to determine what were their values,
perceptions and practices regarding package freshness. Their answers indicated
that they usually place significant value in keeping the packages they use up
to date, principally out of security concerns, but also largely to benefit
from bug fixes and new features. Whenever possible, users prefer to update
packages through the distribution’s official package managers, using the
distribution’s official repositories. This is not always possible, though, as
some packages are either unavailable or outdated in official repositories.
This is most prevalent in the case of proprietary end-user software and some
development tools. Additionally, programming language libraries are often
installed and updated through language-specific package managers.
Unsurprisingly, users perceive packages in rolling release distributions to be
very fresh. On the other hand, Debian Stable and CentOS users perceive it
takes longer for new versions of packages to be made available in the official
repositories, in the order of months. Other distributions are perceived to be
somewhere in between on the “package freshness continuum”.

Preliminary empirical analysis shows that there is some truth to this
perception, with Arch Linux being the most fresh distribution studied and
CentOS the least. In a follow-up work, we will conduct further empirical
analyses in order to quantify the comparative package freshness of Linux
distributions, as well as examine its role as a motivator in user adoption of
distributions.

## Acknowledgment

This research is supported by the Fonds de la Recherche Scientifique – FNRS
under Grants number O.0157.18F-RG43 (Excellence of Science project SECO-
ASSIST) and T.0017.18.

## References

  * [1] Q. Tu _et al._ , “Evolution in open source software: A case study,” in _Proceedings 2000 International Conference on Software Maintenance_. IEEE, 2000, pp. 131–142.
  * [2] G. Robles, J. J. Amor, J. M. Gonzalez-Barahona, and I. Herraiz, “Evolution and growth in large libre software projects,” in _Eighth International Workshop on Principles of Software Evolution (IWPSE’05)_. IEEE, 2005, pp. 165–174.
  * [3] J. Vouillon and R. Di Cosmo, “On software component co-installability,” in _Joint European Software Engineering Conference / Foundations of Software Engineering_ , 2011, pp. 256–266.
  * [4] ——, “Broken sets in software repository evolution,” in _International Conference on Software Engineering_ , 2013, pp. 412–421.
  * [5] M. Claes, T. Mens, R. D. Cosmo, and J. Vouillon, “A historical analysis of Debian package incompatibilities,” in _Working Conference on Mining Software Repositories_ , 2015, pp. 212–223.
  * [6] A. Decan and T. Mens, “What do package dependencies tell us about semantic versioning?” _IEEE Transactions on Software Engineering_ , 2019.
  * [7] S. Raemaekers, A. van Deursen, and J. Visser, “Semantic versioning and impact of breaking changes in the Maven repository,” _Journal of Systems and Software_ , vol. 129, pp. 140 – 158, 2017.
  * [8] C. Artho, K. Suzaki, R. Di Cosmo, R. Treinen, and S. Zacchiroli, “Why do software packages conflict?” in _Working Conference on Mining Software Repositories_ , 2012, pp. 141–150.
  * [9] S. Shawcroft, “Open source watershed: Studying the relationship between linux package and distribution releases,” Ph.D. dissertation, University of Washington, 2009.
  * [10] J. M. Gonzalez-Barahona, G. Robles, M. Michlmayr, J. J. Amor, and D. M. German, “Macro-level software evolution: a case study of a large software compilation,” _Empirical Software Engineering_ , vol. 14, no. 3, pp. 262–285, 2009.
  * [11] R. Nguyen and R. Holt, “Life and death of software packages: an evolutionary study of debian,” in _Proceedings of the 2012 Conference of the Center for Advanced Studies on Collaborative Research_ , 2012, pp. 192–204.
  * [12] J. M. Gonzalez-Barahona, P. Sherwood, G. Robles, and D. Izquierdo, “Technical lag in software compilations: Measuring how outdated a software deployment is,” in _IFIP International Conference on Open Source Systems_. Springer, 2017, pp. 182–192.
  * [13] A. Zerouali, T. Mens, J. Gonzalez-Barahona, A. Decan, E. Constantinou, and G. Robles, “A formal framework for measuring technical lag in component repositories – and its application to npm,” _Journal of Software: Evolution and Process_ , vol. 31, no. 8, 2019.
  * [14] A. Zerouali, T. Mens, G. Robles, and J. M. Gonzalez-Barahona, “On the relation between outdated Docker containers, severity vulnerabilities, and bugs,” in _International Conference on Software Analysis, Evolution and Reengineering_. IEEE, Feb 2019, pp. 491–501.
  * [15] A. Decan, T. Mens, and E. Constantinou, “On the evolution of technical lag in the npm package dependency network,” in _International Conference on Software Maintenance and Evolution_. IEEE, September 2018, pp. 404–414.


# eXmY: A Data Type and Technique for
Arbitrary Bit Precision Quantization

Aditya Agrawal
<EMAIL_ADDRESS>
Matthew Hedlund
Blake Hechtman
<EMAIL_ADDRESS>Google LLC

###### Abstract

eXmY is a novel data type for quantization of ML models. It supports both
arbitrary bit widths and arbitrary integer and floating point formats. For
example, it seamlessly supports 3, 5, 6, 7, 9 bit formats. For a specific bit
width, say 7, it defines all possible formats e.g. e0m6, e1m5, e2m4, e3m3,
e4m2, e5m1 and e6m0. For non-power of two bit widths e.g. 5, 6, 7, we created
a novel encoding and decoding scheme which achieves perfect compression, byte
addressability and is amenable to sharding and vector processing. We
implemented libraries for emulation, encoding and decoding tensors and
checkpoints in C++, TensorFlow, JAX and PAX. For optimal performance, the
codecs use SIMD instructions on CPUs and vector instructions on TPUs and GPUs.
eXmY is also a technique and exploits the statistical distribution of
exponents in tensors. It can be used to quantize weights, static and dynamic
activations, gradients, master weights and optimizer state. It can reduce
memory (CPU DRAM and accelerator HBM), network and disk storage and transfers.
It can increase multi tenancy and accelerate compute. eXmY has been deployed
in production for almost 2 years.

##  1 Introduction

The relentless growth in model size poses significant challenges for model
training, pretraining, finetuing and serving. Large Embedding Models (LEMs)
e.g. DLRM [44] and Large Language Models (LLMs) e.g. PaLM [9], LLaMA [58, 59,
38], GPT-3 [7], have large memory footprint, memory and network bandwidth
requirements, compute requirements, serving latencies, energy consumption and
cost.

Quantization is a proven approach to mitigate these challenges, by reducing
the precision of model weights, master weights, activations, gradients,
optimizer states, and network communication. However, most existing
quantization techniques and hardware rely on conventional power-of-two bit
widths and formats, which may not be ideally suited for preserving model
quality in all use cases.

Previously, ML accelerators e.g. Google TPUs [30, 23, 24], and Nvidia GPUs
[46, 47] added support for int8 and int4 datatypes. More recently, Nvidia H100
[47] and Nvidia GB200 [49] have added support for fp8, fp6 and fp4 datatypes.
Nvidia TensorFloat32 [48] and the OCP [54] fp6 formats e.g. e2m3 and e3m2 are
a step in the direction of supporting non power-of-two bit widths. However,
they do not address the entire problem space. In addition, they do not provide
a bit packing and unpacking scheme to actually reduce the memory footprint and
bandwidth.

Different layers and operations within a model have different sensitivity to
precision, for example, the authors in [39] suggest using e4m3 for weight and
activation tensors, and e5m2 for gradient tensors. In this work, we propose
and advocate the use of flexible, arbitrary bit precision formats which can be
tailored to the specific requirements of each model component e.g. master
weights, training weights, serving weights, network communication etc. Our
contributions are

  * •

A novel datatype which supports arbitrary bit widths and formats.

  * •

A software library to emulate any datatype using existing bfloat16 or float32
datatypes. This enables very fast evaluation of model quality at different
formats and bit widths. The library can be used to quantize weights, master
weights, static and dynamic activations, gradients, optimizer states and
network communication. The library preserves NaNs and Infs for easy debugging.

  * •

Software codecs for packing and unpacking bits into existing datatypes. The
codecs achieve perfect compression, offer byte addressability, works
seamlessly with sharding and are amenable to vector processing on CPUs, GPUs
and TPUs for high performance.

  * •

Discovered a distribution of exponents in ML models and proposed a technique
to exploit the distribution to significantly reduce the number of bits
required by ML models.

##  2 A New Datatype

Table 1: Floating point datatypes. Format | AKA | # Bits | Sign Bit | # Exponent Bits | # Mantissa Bits | Exponent Bias
---|---|---|---|---|---|---
fp32 | e8m23 | 32 | 1 | 8 | 23 | 127
tf32 | e8m10 | 19 | 1 | 8 | 10 | 127
bf16 | e8m7 | 16 | 1 | 8 | 7 | 127
fp16 | e5m10 | 16 | 1 | 5 | 10 | 15
fp8 | e4m3 | 8 | 1 | 4 | 3 | 7
fp8 | e5m2 | 8 | 1 | 5 | 2 | 15
| eXmY | $1+X+Y$ | 1 | $X$ | $Y$ | variable

Over the years, many floating point formats have been proposed. Some of those
have been IEEE standardized e.g. float64, float32 and float16 [40]. Some are
vendor specific e.g. bfloat16 from Google [25] and tensorfloat32 from NVidia
[48]. Others like fp8, fp6, fp4 [54] have been proposed recently by the Open
Compute Project (OCP). Some formats like float32 have only one definition
i.e., 1 sign bit, 8 exponent bits, 23 mantissa bits, exponent bias of $127$,
supports subnormals, NaNs, positive and negative infinities, while, others
like fp8 support multiple formats within the same bit width e.g. e4m3 and
e5m2. Table 1, shows the bit allocation and exponent bias for a few different
data types.

Format

eXmY is a generalization of the floating point format to arbitrary bit widths
and formats. It has 1 sign bit, $X$ exponent bits and $Y$ mantissa bits. For
example, with 7 bits, it defines 7 formats viz. e6m0, e5m1, e4m2, e3m3, e2m4,
e1m5 and e0m6.

When $X=1$, the format becomes linear and equivalent to a symmetric signed
integer format, e.g. e1m2 is equivalent to symmetric int4 and can represent
integers from $[-7,7]$, e1m3 is equivalent to symmetric int5 and can represent
integers from $[-15,15]$ etc. This equivalence enables comparing integer and
floating point formats more easily, for example, their dynamic range and
precision. It also enables implementing integer arithmetic using floating
point hardware.

When $X=0$, the format degenerates to the form (sign, magnitude). Like
floating point numbers, it has a double zero, but it can be instead
interpreted as a 2’s complement number to get an additional encoding. For
example, e0m3 can be used as int4 and represent integers from $[-8,7]$.

Therefore, eXmY can represent signed integers, symmetric signed integers and
floating point numbers. Overall, for bit widths less than and equal to 8, it
defines 36 different formats from e7m0 down to e0m0. For bit widths between 8
and 32 there are dozens of formats e.g. e5m4.

Subnormals

Subnormals, i.e. an exponent value of zero and non zero mantissa, increase the
dynamic range of the representation. eXmY supports subnormals like other
floating point formats.

Rounding

The IEEE 754 standard [40] defines 5 rounding modes viz. roundTiesToEven,
roundTiesToAway, roundTowardPositive, roundTowardNegative and roundTowardZero.
The rounding mode roundTiesToEven, also referred to as, Round To Nearest Even
(RTNE), is the default rounding mode for binary formats. We extended the RTNE
logic in Eigen [26] for rounding from float32 to bfloat16, to arbitrary number
of mantissa bits. We preserve NaNs and Infs during rounding.

NaNs & Infs

Support for NaNs and Infs is optional in eXmY. This is especially important
for serving in sub byte precision, because trained ML model weights do not
have NaNs or Infs.

Exponent Bias

In the IEEE and OCP formats, the exponent bias, the smallest normal and the
normal exponent range are defined by the standard. These values are
interdependent and there is only 1 degree of freedom. For example, in the IEEE
float32 format, the exponent bias is $127$, the smallest normal is $2^{-126}$
and the normal exponent range is $[2^{-126},2^{127}]$. For the OCP E4M3
format, the corresponding values are $7$, $2^{-6}$ and $[2^{-6},2^{8}]$.
However, in eXmY, these values are software defined and is stored as metadata.
For example, in e3m3, with 3 exponent bits, the corresponding values could be
($2$, $2^{-1}$, $[2^{-1},2^{5}]$) or ($-1$, $2^{2}$, $[2^{2},2^{8}]$) i.e. the
value $2^{0}=1$ is not even in the normal exponent range in the second
example.

Metadata

Since the byte and sub-byte formats have limited dynamic range, eXmY, OCP
formats [54], conventional int8 and int4 quantization schemes, maintain some
metadata. Typically, with int8 and int4 quantization, the metadata is a
bfloat16 or float32 scaling factor. In the OCP formats, the metadata is an
8-bit power-of-2 scaling factor. Its format is the same as the 8-bit exponent
field of the IEEE float32 format. In eXmY, the metadata is the value of the
maximum biased exponent, an 8 bit value.

The maximum biased exponent can be determined before or after rounding to the
appropriate number of mantissa bits. An additional bfloat16 or float32 scaling
factor can also be maintained.

Block Size

The OCP formats define a block size of $32$, i.e. the metadata is shared
between $32$ elements. eXmY does not define or constrain the size or shape of
the block. A block can be a tensor, a row, a column, a sub row or even a 2D
tile. As is obvious, more metadata improves model quality at the expense of
storage. In general, we have observed that for LLM serving, e3m2 and e3m1
require only one metadatum per row, while e2m1 and e1m2 benefit from smaller
block sizes.

###  2.1 Emulation

Just like we can emulate int5 or int7 using an int8 datatype, likewise, we can
emulate any eXmY format using bfloat16, if $X\leq 8$ and $Y\leq 7$, or using
fp16, if $X\leq 5$ and $Y\leq 10$, or using float32, if $X\leq 8$ and $Y\leq
23$. We preserve NaNs and Infs during emulation.

Fig. 1 shows a scatter plot of the original values vs. values emulated with
e2m1 at block size 16 and at three different schemes viz. maximum exponent
before rounding, maximum exponent after rounding, and float scaling with
maximum exponent of 127. Note that the same input value can either be (a)
saturated to the largest normal, (b) rounded to the appropriate number of
mantissa bits, (c) considered a subnormal, or (d) flushed to zero, depending
on its relative value in the block.

The first two plots have a staircase pattern with one step between every power
of two. The scheme maximum exponent after rounding is useful at small block
sizes to prevent excessive truncation of the largest value in the block. For
example, in the first scheme, 3.9 either rounds down to 3.0 or rounds up to
4.0, while, in the second scheme, it always rounds up to 4.0. The float
scaling scheme captures the largest value in the block accurately.

Figure 1: Emulation using e2m1 with different schemes.

###  2.2 Codecs: Encoder & Decoder

Current processors provide only a few compute data types e.g. float32,
bfloat16, int8, int4, OCP e4m3 etc., however, eXmY supports dozens of formats.
Therefore, we need software routines or hardware instructions to encode and
decode from eXmY data types.

The encoding and decoding can be done offline or on the fly. For example,
trained weights and static activations (feature maps) can be encoded offline
and is not performance critical. However, decoding weights during serving or
encoding and decoding the dynamic activations and gradients before and after
network communication is performance critical. The codecs have two components:

####  2.2.1 Type Conversion: Float $\longleftrightarrow$ eXmY + Metadata

In this step, we convert a float format to an eXmY format and store it an 8,
16 or 32 bit container and vice versa. The metadata i.e., maximum exponent, is
maintained separately. For example, we convert a bfloat16 tensor of shape
$(R,C)$ to an int8 tensor of shape $(R,C)$ containing e3m3 values, and an int8
tensor of shape $(R,1)$ containing the metadata.

####  2.2.2 Bit Packing & Unpacking: Power-of-2 Decomposition

Consider an array, where each element is a 7-bit eXmY datatype e.g., e3m3.
Fig. 2, shows the scheme for packing and unpacking an array of shape $(8,1)$
with 7-bit elements. Before packing and after unpacking, the elements are held
in an int8 container as shown in the figure. We decompose the bits into
_power-of-2_ segments i.e., $7=4+2+1$. We pack eight 4-bit elements into an
int32 container, eight 2-bit elements into an int16 container, and eight 1-bit
elements into an int8 container, as shown on the right. Overall, an array of
shape $(8R,C)$ gets packed into 3 arrays of int32, int16 and int8, each of
shape $(R,C)$. There are many advantages of this scheme.

  * •

Uses existing storage datatypes e.g. int32, int16 and int8.

  * •

Independent of the data format e.g., the 7-bit format could be either e4m2 or
e3m3.

  * •

Achieves perfect compression i.e., 8 elements of 7-bits use exactly
$32+16+8=56$ bits.

  * •

Works for all arbitrary bit widths. For example, an array of shape $(8R,C)$
containing 5-bit elements ($5=4+1$) can be packed into 2 arrays of int32
andint8, each of shape $(R,C)$.

  * •

Amenable to SIMD and vector processing on current CPUs, GPUs and TPUs.

  * •

The array can be sharded along rows or columns, before or after packing, and
each shard can be independently reconstructed.

  * •

Can be modified to pack along columns i.e., an array of shape $(R,8C)$ can be
packed into multiple arrays of shape $(R,C)$.

The only constraint of the scheme is that the number of rows or columns is a
multiple of 8, which is almost always true in ML models.

7 bits8 elementsint32int16int8 Figure 2: Bit packing and unpacking for 7-bit
wide elements.

##  3 Technique

###  3.1 Exponent Distribution

Both float32 and bfloat16 use 8 exponent bits, i.e., they can encode 256
exponent values. Also both formats have an exponent bias of 127 i.e., an
exponent of 1 ($2^{1}$) is stored as $127+1=128$. $0$ has an exponent of zero.

The plot below shows the histogram of the exponent values in one of the PaLM-2
layers [1]. The X-axis shows the biased exponent value which ranges from
$[0,255]$. The X2-axis on top shows the corresponding floating point values.
The Y-axis shows the histogram on a $log_{10}$ scale. The exponent
distribution shifts a little but has the same shape for both weigths and
activations.

Figure 3: Histogram of the exponent values.

There are multiple observations from this plot:

  * •

There are no absolute zeros in the value distribution. However, if the tensor
is zero initialized, as is the case for some large embedding models, we do
observe some absolute zeros.

  * •

The left side of the distribution is linear in the log scale. For example, the
number of values with exponent 101 is two times the number of values with
exponent 100. The number of values with exponent 120 is $2^{20}\approx
10^{6}$, times the number of values with exponent 100. This implies that the
values are uniformly distributed on the left side.

  * •

The distribution reaches a peak and then drops sharply.

  * •

The fraction of values with a large magnitude, e.g. $[2,16]$ is very small
$\approx$ less than 1%. This is because ML models typically, but not always,
use weight clipping and weight regularization. Models which don’t use weight
clipping or regularization have a higher fraction.

  * •

Only a small range, typically $\\{0,[80-140]\\}$, of biased exponents are
used. This implies we need only 6 bits, instead of 8 bits, for lossless
encoding of exponents.

  * •

The fraction of values with a small magnitude, e.g. $(0,2^{-10})$ i.e.
exponents in the range $[0,116]$ is very small $\approx 0.11\%$.

The last observation is the most important. In the above distribution, if we
flush values with smaller exponents, i.e. $[0,116]$ to zero (exponent of 0)
and retain only the top 15 exponents, i.e. $[117,131]$, then we need only 4
bits to encode the exponents $\\{0,[117,131]\\}$. The hypothesis is that
flushing the exponent tail to zero will have minimal impact on model quality.

Note that using subnormals and more metadata e.g. _max exponent_ per row,
instead of per tensor, significantly reduces the fraction of values flushed to
zero and improves model quality. We found that e4mY with per tensor metadata
and e3mY with per row or column metadata is quality neutral for a wide variety
of Large Embedding Models (LEMs) and Large Language Models (LLMs). e2mY
generally requires metadata at finer granularity. See quality evaluation in
Section 6.

###  3.2 #Mantissa Bits vs Quality

Table 2 shows the model quality of the PaLM 2 S model [1], for a few LLM
datasets as we reduce the number of mantissa bits of the Feed Forward Networks
(FFN) weights, using Post Training Quantization (PTQ). The baseline is
bfloat16, i.e. e8m7. We can observe that the model quality is fairly neutral
even with just 1 or 2 mantissa bits. However, the quality drops significantly
with zero mantissa bits i.e. only power of 2 numbers.

Table 2: PaLM 2 S quality vs number of mantissa bits. format | base | e8m6 | e8m5 | e8m4 | e8m3 | e8m2 | e8m1 | e8m0
---|---|---|---|---|---|---|---|---
hellaswag | 64.45 | 64.59 | 64.56 | 64.49 | 64.42 | 64.49 | 64.02 | 64.30
lambada | 84.15 | 84.26 | 84.26 | 83.95 | 84.03 | 83.60 | 82.77 | 65.07
squadv2 | 75.46 | 75.31 | 75.46 | 75.38 | 75.47 | 75.09 | 73.32 | 71.53
triviaqa | 77.21 | 77.28 | 77.26 | 77.23 | 77.29 | 76.79 | 75.74 | 70.60
webqs | 23.47 | 23.33 | 23.38 | 23.62 | 23.43 | 23.23 | 21.95 | 20.57

Combining the observations in this and the previous section, we found that
e3m1 with per row metadata is fairly quality neutral for LLMs. e2m1 and e1m2
benefit from metadata at finer granularity. See quality evaluation in Section
6.

##  4 Applications

eXmY can be used to (a) Quantize weights, static and dynamic activations and
gradients (b) Quantize master weights and optimizer state (c) Accelerate
compute (d) Increase multi tenancy (e) Reduce memory transfers (f) Reduce
network (PCIe, data center network) transfers (g) Reduce disk storage and disk
transfer.

eXmY can be used for both Post Training Quantization (PTQ) and Quantization
Aware Training (QAT). It can be used with both symmetric and affine
quantization schemes. It can be combined with other techniques e.g. sparsity
and lossless compression algorithms e.g. Zstandard [13]. Since eXmY is also a
datatype it can be used with other quantization recipes and techniques e.g.
HAWQ [19], QLoRA [16], OPTQ [21], SmoothQuant [63] etc.

eXmY allows choosing the number of exponent bits, mantissa bits, and block
size on a per tensor basis and hence enables a gradual trade off between model
quality and compression. The emulation and codecs work on all existing CPUs,
GPUs and TPUs, but can benefit from hardware support for conversions and bit
packing and unpacking.

##  5 Limitations and Considerations

The eXmY datatype itself has no limitations. During serving, there are no NaNs
or Infs and all encodings have finite values. During training with eXmY
emulation, NaNs and Infs are preserved out of band and hence all eXmY values
are still finite. Training with true eXmY encoded values requires allocating
an encoding(s) for these special values and has not been discussed in this
paper.

The eXmY technique works best for PTQ of weights when models use weight
regularization, weight clipping etc., such that the weights have an exponent
distribution as shown in Fig. 3. When those techniques are not used, there is
a larger fraction of values to the right of the peak and that requires using a
format with a bigger dynamic range e.g. e4m3 instead of e3m4.

Based on the exponent distribution, we can make educated guesses about the
format to use. However, the impact on model quality needs to be measured.
Finally, the acceptable change (drop) in model quality with quantization
depends on the trade off between revenue impact and cost savings.

##  6 Quality Evaluation

We evaluated eXmY on many open source models e.g. ResNet [28], Transformer
[60], BERT [17], as well as many internal vision, ranking, recommendation,
Large Embedding Models (LEMs) and Large Language Models (LLMs). In this
section, we show the quality evaluation on the PaLM 2 S model [1] using the
following datasets: Adversarial NLI (ANLI) [45], ARC [12], BoolQ [10], CB,
COPA [53], COQA, DROP [20], HellaSwag [66], LAMBADA [50], Natural Questions
[32], OpenBookQA [41], PIQA [4], QuAC [8], RACE [33], ReCoRD [67], RTE, SQuAD
v2 [52], StoryCloze [42], TriviaQA [29], TyDi QA [11], WebQuestions [3], WiC
[51], Winograd [34], and WinoGrande [55].

The left half of Table 3, shows the scores when all the Feed Forward Network
(FFN) weights are post training quantized to e3m4, e3m3, e3m2, e3m1, e3m0, and
e2m1 respectively. The attention layers are always quantized to e3m4. The
block size is the length of the row in the weight matrix. The scheme is
maximum exponent before rounding. There are multiple observations from the
table:

Table 3: PaLM 2 S quality at different eXmY formats and block sizes. format | base | e3m4 | e3m3 | e3m2 | e3m1 | e3m0 | e2m1 | e2m1 | e2m1 | e2m1 | e2m1
---|---|---|---|---|---|---|---|---|---|---|---
block_size |  | row | row | row | row | row | row | 512 | 256 | 128 | 64
anlir1 | 53.00 | 53.90 | 54.10 | 53.80 | 54.30 | 52.40 | 51.30 | 52.90 | 53.80 | 53.70 | 53.70
anlir2 | 49.00 | 49.30 | 49.10 | 48.60 | 49.30 | 46.90 | 47.20 | 45.40 | 46.80 | 47.60 | 48.20
anlir3 | 52.75 | 53.08 | 53.42 | 53.17 | 53.92 | 53.92 | 52.08 | 53.00 | 52.50 | 51.75 | 52.75
arcchallenge | 56.06 | 56.57 | 56.74 | 56.74 | 55.46 | 54.69 | 52.13 | 54.86 | 55.03 | 55.63 | 55.63
arceasy | 84.93 | 84.89 | 84.89 | 85.06 | 84.05 | 84.13 | 78.96 | 82.74 | 82.87 | 83.21 | 83.54
boolq | 89.08 | 88.81 | 88.93 | 88.90 | 88.96 | 86.88 | 79.08 | 85.57 | 87.80 | 88.29 | 88.13
cb | 87.50 | 87.50 | 85.71 | 91.07 | 83.93 | 87.50 | 76.79 | 82.14 | 83.93 | 85.71 | 85.71
copa | 89.00 | 88.00 | 87.00 | 87.00 | 89.00 | 88.00 | 89.00 | 86.00 | 87.00 | 87.00 | 88.00
coqa | 63.06 | 63.21 | 63.22 | 62.92 | 62.81 | 59.64 | 61.44 | 62.73 | 62.51 | 62.29 | 62.73
drop | 54.60 | 54.64 | 54.56 | 54.24 | 53.87 | 50.06 | 51.68 | 53.32 | 53.33 | 53.75 | 53.97
hellaswag | 64.45 | 64.53 | 64.33 | 64.54 | 64.01 | 64.20 | 62.32 | 64.02 | 63.68 | 63.68 | 63.13
lambada | 84.15 | 84.30 | 84.34 | 83.58 | 83.27 | 65.07 | 75.57 | 79.64 | 80.38 | 82.50 | 83.17
eue19_defr | 36.17 | 35.96 | 35.91 | 35.58 | 35.71 | 34.17 | 31.62 | 34.98 | 35.10 | 35.55 | 35.45
eue19_frde | 26.79 | 26.66 | 26.09 | 24.93 | 27.31 | 26.62 | 21.04 | 25.78 | 25.94 | 26.11 | 26.53
wmt14_enfr | 44.89 | 44.96 | 45.07 | 45.06 | 44.44 | 41.79 | 41.42 | 43.35 | 43.35 | 43.69 | 43.97
wmt14_fren | 44.96 | 44.85 | 45.26 | 44.80 | 44.74 | 41.29 | 41.56 | 43.92 | 44.41 | 44.56 | 44.48
wmt16_deen | 48.37 | 48.56 | 48.53 | 48.29 | 48.02 | 45.24 | 44.59 | 47.70 | 47.82 | 48.10 | 47.90
wmt16_ende | 39.44 | 39.37 | 39.32 | 39.13 | 38.75 | 34.20 | 35.65 | 37.73 | 38.16 | 37.82 | 38.42
wmt16_enro | 32.63 | 32.66 | 32.72 | 32.50 | 32.76 | 31.70 | 31.53 | 32.46 | 32.46 | 32.72 | 32.68
wmt16_roen | 46.62 | 46.59 | 46.50 | 46.62 | 46.27 | 44.84 | 44.08 | 45.35 | 45.96 | 45.87 | 45.92
wmt19_enkk | 8.36 | 8.48 | 8.24 | 8.66 | 8.53 | 5.82 | 7.71 | 7.88 | 7.09 | 7.01 | 7.61
wmt19_enzh | 5.28 | 5.12 | 5.20 | 5.06 | 4.99 | 4.46 | 5.90 | 5.38 | 5.48 | 4.86 | 4.87
wmt19_kken | 31.15 | 31.23 | 31.41 | 31.16 | 31.07 | 28.99 | 26.46 | 29.97 | 30.32 | 30.71 | 30.92
wmt19_zhen | 32.76 | 32.63 | 32.47 | 32.43 | 31.57 | 28.12 | 28.43 | 30.56 | 30.88 | 30.81 | 31.54
nqs | 27.92 | 28.06 | 27.78 | 27.29 | 26.32 | 22.35 | 20.00 | 24.71 | 24.85 | 24.46 | 26.04
openbookqa | 47.80 | 47.60 | 47.80 | 48.40 | 46.40 | 44.60 | 42.40 | 47.00 | 46.20 | 45.60 | 47.20
piqa | 81.01 | 81.18 | 81.18 | 81.23 | 80.85 | 80.36 | 77.97 | 80.79 | 80.69 | 80.63 | 80.85
quac | 23.46 | 23.43 | 23.42 | 23.39 | 22.83 | 19.87 | 20.51 | 22.49 | 22.80 | 22.70 | 22.57
raceh | 48.31 | 48.28 | 48.68 | 48.68 | 48.74 | 48.48 | 47.20 | 49.06 | 48.54 | 48.91 | 48.80
racem | 64.83 | 64.97 | 65.67 | 65.04 | 65.04 | 63.79 | 63.02 | 64.48 | 64.55 | 64.76 | 64.69
record | 92.10 | 92.22 | 92.02 | 92.15 | 91.93 | 89.44 | 89.34 | 91.20 | 91.37 | 91.73 | 91.80
rte | 77.26 | 77.62 | 77.26 | 78.34 | 77.98 | 75.45 | 79.42 | 77.98 | 77.62 | 77.98 | 77.98
squadv2 | 75.46 | 75.63 | 75.63 | 75.25 | 73.52 | 71.70 | 75.41 | 76.18 | 74.98 | 75.19 | 74.73
storycloze | 81.88 | 81.83 | 82.36 | 82.26 | 81.51 | 82.31 | 78.67 | 81.93 | 81.56 | 81.56 | 81.13
triviaqa | 77.21 | 77.33 | 77.43 | 76.77 | 75.82 | 71.01 | 67.43 | 73.90 | 74.30 | 74.68 | 75.77
tydiaqa | 17.31 | 17.33 | 17.14 | 17.02 | 16.47 | 14.24 | 14.08 | 16.27 | 16.15 | 16.15 | 16.09
webqs | 23.47 | 23.47 | 23.18 | 23.03 | 21.65 | 20.62 | 17.27 | 22.24 | 21.21 | 21.46 | 22.24
wic | 51.10 | 51.25 | 50.47 | 53.61 | 50.94 | 50.31 | 50.16 | 50.00 | 50.31 | 50.63 | 50.16
winograd | 84.98 | 84.98 | 85.71 | 84.25 | 84.25 | 82.78 | 79.12 | 84.62 | 82.78 | 84.98 | 83.15
winogrande | 77.35 | 77.82 | 77.03 | 78.14 | 77.19 | 75.22 | 69.46 | 73.40 | 74.11 | 76.40 | 75.37
wsc273 | 84.62 | 85.35 | 84.98 | 83.88 | 84.62 | 82.05 | 77.66 | 83.15 | 82.05 | 84.98 | 81.32

  * •

Overall, LLMs hold their quality very well with simple PTQ of weights down to
e3m1 even with per-row metadata and without requiring any advanced techniques
like SmoothQuant [63], OPTQ [21], ZeroQuant [65] etc.

  * •

The quality does not decrease monotonically as we reduce the number of
exponent and/or mantissa bits. For example, for OpenBookQA and PIQA, the
quality with e3m2 is better than bfloat16, which is e8m7. We suspect this is
due to the opposing effects of quantization and regularization.

  * •

Some datasets e.g. HellaSwag are very resilient to quantization, while others
e.g. LAMBADA are more sensitive, i.e. the choice of the quantization format is
dataset dependent.

  * •

The quality drop is significant at 4 bit formats e.g. e3m0 and e2m1 at large
block sizes.

The quality of e2m1 improves by reducing the block size. The right half of
Table 3, shows the scores when the block size is reduced from row to 512 and
then to 64 in powers of 2. We can observe that for sensitive datasets like
LAMBADA, the quality increases monotonically as we decrease the block size.

##  7 Related Work

Posits [27, 36] are an alternative way of representing real numbers. They
offer a good trade-off between dynamic range and accuracy, encounter fewer
exceptions, and have tapered precision i.e. numbers near $\pm$1 have more
precision, while very big and very small numbers have less. Other floating
point formats have also been proposed e.g. Logarithmic numbers [14] and
NormalFloat4 [16] which targets normally distributed weights.

Numerous studies and techniques compare and use different data types in
various settings, such as post training quantization (PTQ), quantization aware
training (QAT) and fully quantized training (FQT). For quantized inference,
multiple industry and academic white papers highlight the overall benefits and
general approaches to int8 (sometimes even int4) quantization [62, 43, 22]
exploring quantization granularity, scaling methods, initialization
techniques, and data formats.

For LLM quantization, a plethora of techniques have emerged such as one-shot
PTQ techniques with layer-wise optimizations [21], optimization free
techniques which leverage robustness of data types (fp8) [31], and 4 bit
techniques with searches for exponents bits and clipping range [35]. After
analyzing the causes of quality degradation in LLM quantization, various
authors have identified outlier behaviour to be problematic and proposed
various solutions, such as offline transformation of weights to absorb
outliers [63], channel-wise shifting and scaling [61], rotation of hidden
state matrices [2], modifications of the attention mechanism [6], and mixed-
precision matrix decomposition [15].

To combat model quality degradation at lower bit-widths, some previous works
propose mixed precision approaches which keep sensitive layers in higher
precision, whereby the sensitivity is usually approximated through a Hessian
[19, 18, 64, 57]. Alternatively, to improve quality other works incorporate
quantization consideration into training (QAT), for example through optimizing
clipping scalars [56] or data-free distillation method based on outputs of a
pretrained model [35].

Extending quantization to training (FQT), QLoRA [16] reduces the memory
requirements for LLM finetuning by quantizing the weights of the frozen
pretrained model to 4 bits. Going even further [37] quantize weights,
activations, errors, gradients, and the master copy of the weights during
training and achieve SOTA through various data sets and models utilizing loss
scaling method to augment the reduced subnormal range and stochastic rounding.
Attempting to simplify training with FP8 [5] present unit scaling, a paradigm
which yields unit variance for weights, activations, and gradients at
initialization. This approach works without quality degradation across
multiple optimizers and models.

##  8 Conclusion

In this work, we described eXmY, a new data type and technique for
quantization of ML models. It can represent arbitrary bit width signed
integers, symmetric signed integers and floating point numbers. It supports
subnormals and arbitrary block shapes and sizes.

We described a novel bit packing scheme which achieves perfect compression
using existing storage data types. It works for all arbitrary bit widths and
is amenable to vector processing on all existing hardware. The scheme offers
byte addressability and works seamlessly with array sharding. We implemented
libraries for emulation, encoding and decoding tensors in multiple frameworks.

We discovered the distribution of exponents in ML models. We described a
technique to exploit it and significantly reduce the number of bits required
by the model while retaining model quality. The technique can be used to
quantize master weights, training weights, serving weights, static and dynamic
activations, gradients and network communication. This reduces CPU RAM
footprint and bandwidth, accelerator RAM (HBM) footprint and bandwidth, PCIe
and network latency, disk I/O and increases multi-tenancy. With hardware
support the technique can also be used for compute acceleration.

eXmY has been deployed in production by multiple teams. We have found many
interesting applications and hope the community at large will embrace
arbitrary bit widths and formats to develop novel techniques and applications.

## Acknowledgments and Disclosure of Funding

Abdullah Rashwan, Afroz Mohiuddin, Alex Tomala, Andy Chu, Anselm Levskaya,
Bing-Rong Lin, Chen Chen, Chris Waterson, Clemens Schaefer, Dar-Shyang Lee,
David Majnemer, Diego Caballero, Eric Chang, Erica Moreira, Eugene Zhulenev,
Grant Wang, Gil Tabak, Guangda Lai, Jacques Liao, Jaideep Singh, Jayant
Madhavan, Jomy Alappattu, Jon Clark, Kirill Borozdin, Kirk Sanders, Lili Hu,
Marissa Ikonomidis, Matthew Fahrbach, Michael Mangan, Ming Liu, Nand Dalal,
Naveen Kumar, Navid Lambert-Shirzad, Nathan Lintz, Orhan Firat, Pierre-
François Laquerre, Pidong Wang, Phuong Dao, Pooja Aggarwal, Qi Lyu, Qi Wu,
Rahul Nagarajan, Rajkumar Samuel, Rakshith Reddy Polireddy, Reza Sadoddin,
Rigel Swavely, Robin Sabhnani, Rohan Anil, Ryan Doherty, Sencer Selcuk,
Shaolin Qu, Shen Wu, Shicheng Xu, Shruti Gupta, Srikanth Dwarakanath, Tao Yu,
Tom Jablin, Victor Akabutu, Woohyun Han, Yang Li, Yiwen Deng, Yuan Huang,
Zongwei Zhou.

## References

  * Anil et al. [2023] R. Anil, A. M. Dai, et al.  PaLM 2 Technical Report, 2023.  URL https://arxiv.org/abs/2305.10403.
  * Ashkboos et al. [2024] S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman.  Quarot: Outlier-free 4-bit inference in rotated llms.  _arXiv preprint arXiv:2404.00456_ , 2024.
  * Berant et al. [2013] J. Berant, A. Chou, R. Frostig, and P. Liang.  Semantic Parsing on Freebase from Question-Answer Pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , Oct. 2013.  URL https://aclanthology.org/D13-1160.
  * Bisk et al. [2020] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi.  PIQA: Reasoning about Physical Commonsense in Natural Language.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020_ , Feb. 2020.  URL https://doi.org/10.1609/aaai.v34i05.6239.
  * Blake et al. [2023] C. Blake, D. Orr, and C. Luschi.  Unit scaling: Out-of-the-box low-precision training.  In _International Conference on Machine Learning_ , pages 2548–2576. PMLR, 2023.
  * Bondarenko et al. [2024] Y. Bondarenko, M. Nagel, and T. Blankevoort.  Quantizable transformers: Removing outliers by helping attention heads do nothing.  _Advances in Neural Information Processing Systems_ , 2024.
  * Brown et al. [2020] T. B. Brown, B. Mann, et al.  Language Models are Few-Shot Learners, 2020.  URL https://arxiv.org/abs/2005.14165.
  * Choi et al. [2018] E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. Zettlemoyer.  QuAC : Question Answering in Context.  _CoRR_ , 2018.  URL http://arxiv.org/abs/1808.07036.
  * Chowdhery et al. [2022] A. Chowdhery, S. Narang, et al.  PaLM: Scaling Language Modeling with Pathways, 2022.  URL https://arxiv.org/abs/2204.02311.
  * Clark et al. [2019] C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova.  BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions.  _CoRR_ , 2019.  URL http://arxiv.org/abs/1905.10044.
  * Clark et al. [2020] J. H. Clark, E. Choi, M. Collins, D. Garrette, T. Kwiatkowski, V. Nikolaev, and J. Palomaki.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages.  _CoRR_ , 2020.  URL https://arxiv.org/abs/2003.05002.
  * Clark et al. [2018] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord.  Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.  _CoRR_ , 2018.  URL http://arxiv.org/abs/1803.05457.
  * Collet and Kucherawy [2021] Y. Collet and M. Kucherawy.  RFC 8878: Zstandard Compression and the application/zstd Media Type, 2021.  URL https://dl.acm.org/doi/10.17487/RFC8878.
  * Dally [2021] B. Dally.  Logarithmic Numbers and Asynchronous Accumulators The Future of DL Chips.  _MLSys_ , April 2021.  URL https://media.mlsys.org/Conferences/MLSYS2021/Slides/DL_Chips_ML_Sys_0421.pdf.
  * Dettmers et al. [2022] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer.  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.  _Advances in Neural Information Processing Systems_ , 2022.  URL https://arxiv.org/abs/2208.07339.
  * Dettmers et al. [2023] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer.  QLoRA: Efficient Finetuning of Quantized LLMs, 2023.  URL https://arxiv.org/abs/2305.14314.
  * Devlin et al. [2018] J. Devlin, M. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _CoRR_ , 2018.  URL http://arxiv.org/abs/1810.04805.
  * Dong et al. [2019a] Z. Dong, Z. Yao, Y. Cai, D. Arfeen, A. Gholami, M. W. Mahoney, and K. Keutzer.  HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks, 2019a.  URL https://arxiv.org/abs/1911.03852.
  * Dong et al. [2019b] Z. Dong, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer.  HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision, 2019b.  URL https://arxiv.org/abs/1905.03696.
  * Dua et al. [2019] D. Dua, Y. Wang, et al.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , June 2019.  URL https://aclanthology.org/N19-1246.
  * Frantar et al. [2023] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh.  OPTQ: Accurate Quantization for Generative Pre-trained Transformers.  In _International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=tcbBPnfwxS.
  * Gholami et al. [2022] A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer.  A survey of quantization methods for efficient neural network inference.  In _Low-Power Computer Vision_ , pages 291–326. Chapman and Hall/CRC, 2022.
  * Google [a] Google.  Google TPU v5e, a.  URL https://cloud.google.com/tpu/docs/v5e.
  * Google [b] Google.  Google TPU v5p, b.  URL https://cloud.google.com/tpu/docs/v5p-training.
  * Google [2019] Google.  BFloat16: The secret to high performance on Cloud TPUs, 2019.  URL https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus.
  * [26] G. Guennebaud, B. Jacob, et al.  Eigen v3.  URL https://gitlab.com/libeigen/eigen/-/blob/7e655c9a5d3f172e06f0ee0380b52cedc583c24f/Eigen/src/Core/arch/Default/BFloat16.h.
  * Gustafson and Yonemoto [2017] J. L. Gustafson and I. Yonemoto.  Beating Floating Point at its Own Game: Posit Arithmetic, 2017.  URL http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf.
  * He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun.  Deep Residual Learning for Image Recognition.  _CoRR_ , 2015.  URL http://arxiv.org/abs/1512.03385.
  * Joshi et al. [2017] M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , July 2017.  URL https://aclanthology.org/P17-1147.
  * Jouppi et al. [2023] N. P. Jouppi, G. Kurian, et al.  TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings, 2023.  URL https://arxiv.org/abs/2304.01433.
  * Kuzmin et al. [2024] A. Kuzmin, M. V. Baalen, Y. Ren, M. Nagel, J. Peters, and T. Blankevoort.  FP8 Quantization: The Power of the Exponent, 2024.  URL https://arxiv.org/abs/2208.09225.
  * Kwiatkowski et al. [2019] T. Kwiatkowski, J. Palomaki, et al.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ , 2019\.  URL https://aclanthology.org/Q19-1026.
  * Lai et al. [2017] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy.  RACE: Large-scale ReAding Comprehension Dataset From Examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , Sept. 2017.  URL https://aclanthology.org/D17-1082.
  * Levesque et al. [2012] H. J. Levesque, E. Davis, and L. Morgenstern.  The Winograd Schema Challenge.  In _Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning_ , 2012.  URL https://cdn.aaai.org/ocs/4492/4492-21843-1-PB.pdf.
  * Liu et al. [2023] S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng.  Llm-fp4: 4-bit floating-point quantized transformers.  _arXiv preprint arXiv:2310.16836_ , 2023.
  * Mallasén et al. [2022] D. Mallasén, R. Murillo, et al.  PERCIVAL: Open-Source Posit RISC-V Core With Quire Capability.  _IEEE Transactions on Emerging Topics in Computing_ , 2022.  URL https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9817027.
  * Mellempudi et al. [2019] N. Mellempudi, S. Srinivasan, D. Das, and B. Kaul.  Mixed precision training with 8-bit floating point.  _arXiv preprint arXiv:1905.12334_ , 2019.
  * Meta [2024] Meta.  LLaMA 3, 2024.  URL https://ai.meta.com/blog/meta-llama-3/.
  * Micikevicius et al. [2022] P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu.  FP8 Formats for Deep Learning, 2022.  URL https://arxiv.org/abs/2209.05433.
  * Microprocessor Standards Committee [2019] Microprocessor Standards Committee.  IEEE Standard for Floating-Point Arithmetic.  _IEEE Std 754-2019 (Revision of IEEE 754-2008)_ , 2019.  URL https://standards.ieee.org/ieee/754/6210/.
  * Mihaylov et al. [2018] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , Oct.-Nov. 2018.  URL https://aclanthology.org/D18-1260.
  * Mostafazadeh et al. [2016] N. Mostafazadeh, N. Chambers, et al.  A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , June 2016.  URL https://aclanthology.org/N16-1098.
  * Nagel et al. [2021] M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. Van Baalen, and T. Blankevoort.  A white paper on neural network quantization.  _arXiv preprint arXiv:2106.08295_ , 2021.
  * Naumov et al. [2019] M. Naumov, D. Mudigere, et al.  Deep Learning Recommendation Model for Personalization and Recommendation Systems, 2019.  URL https://arxiv.org/abs/1906.00091.
  * Nie et al. [2020] Y. Nie, A. Williams, et al.  Adversarial NLI: A New Benchmark for Natural Language Understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , July 2020.  URL https://aclanthology.org/2020.acl-main.441.
  * NVIDIA [a] NVIDIA.  NVIDIA A100 Tensor Core GPU Architecture, a.  URL https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf.
  * NVIDIA [b] NVIDIA.  NVIDIA H100 Tensor Core GPU Architecture, b.  URL https://resources.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper.
  * NVIDIA [2020] NVIDIA.  TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x, 2020.  URL https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/.
  * NVIDIA [2024] NVIDIA.  NVIDIA Blackwell Architecture, 2024.  URL https://resources.nvidia.com/en-us-blackwell-architecture.
  * Paperno et al. [2016] D. Paperno, G. Kruszewski, et al.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Aug. 2016.  URL https://aclanthology.org/P16-1144.
  * Pilehvar and Camacho-Collados [2019] M. T. Pilehvar and J. Camacho-Collados.  WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations, 2019.  URL https://arxiv.org/abs/1808.09121v3.
  * Rajpurkar et al. [2018] P. Rajpurkar, R. Jia, and P. Liang.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , July 2018.  URL https://arxiv.org/abs/1806.03822.
  * Roemmele et al. [2011] M. Roemmele, C. Bejan, and A. Gordon.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  _AAAI Spring Symposium - Technical Report_ , 2011.  URL https://cdn.aaai.org/ocs/2418/2418-10878-1-PB.pdf.
  * Rouhani et al. [2023] B. D. Rouhani, N. Garegrat, et al.  OCP Microscaling Formats (MX) Specification, Sep 2023.  URL https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf.
  * Sakaguchi et al. [2021] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi.  WinoGrande: An Adversarial Winograd Schema Challenge at Scale, Aug. 2021\.  URL https://doi.org/10.1145/3474381.
  * Sakr et al. [2022] C. Sakr, S. Dai, R. Venkatesan, B. Zimmer, W. Dally, and B. Khailany.  Optimal clipping and magnitude-aware differentiation for improved quantization-aware training.  In _International Conference on Machine Learning_ , pages 19123–19138. PMLR, 2022.
  * Schaefer et al. [2023] C. J. Schaefer, N. Lambert-Shirzad, X. Zhang, C. Chou, T. Jablin, J. Li, E. Guo, C. Stanton, S. Joshi, and Y. E. Wang.  Augmenting hessians with inter-layer dependencies for mixed-precision post-training quantization.  _arXiv preprint arXiv:2306.04879_ , 2023.
  * Touvron et al. [2023a] H. Touvron, T. Lavril, et al.  LLaMA: Open and Efficient Foundation Language Models, 2023a.  URL https://arxiv.org/abs/2302.13971.
  * Touvron et al. [2023b] H. Touvron, L. Martin, et al.  Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023b.  URL https://arxiv.org/abs/2307.09288.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, et al.  Attention Is All You Need.  _CoRR_ , 2017.  URL http://arxiv.org/abs/1706.03762.
  * Wei et al. [2023] X. Wei, Y. Zhang, Y. Li, X. Zhang, R. Gong, J. Guo, and X. Liu.  Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.  _arXiv preprint arXiv:2304.09145_ , 2023.
  * Wu et al. [2020] H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius.  Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation, 2020.  URL https://arxiv.org/abs/2004.09602.
  * Xiao et al. [2024] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han.  SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, 2024.  URL https://arxiv.org/abs/2211.10438.
  * Yao et al. [2021] Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y. Wang, M. W. Mahoney, and K. Keutzer.  HAWQV3: Dyadic Neural Network Quantization, 2021.  URL https://arxiv.org/abs/2011.10680.
  * Yao et al. [2022] Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He.  ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers, 2022.  URL https://arxiv.org/abs/2206.01861.
  * Zellers et al. [2019] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi.  HellaSwag: Can a Machine Really Finish Your Sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , July 2019.  URL https://aclanthology.org/P19-1472.
  * Zhang et al. [2018] S. Zhang, X. Liu, J. Liu, J. Gao, K. Duh, and B. V. Durme.  ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension.  _CoRR_ , 2018.  URL http://arxiv.org/abs/1810.12885.