# Automating SBOM Generation with Zero-Shot Semantic Similarity

Devin Pereira, Christopher Molloy, Sudipta Acharya, Steven H.H. Ding
School of Computing
Queen’s University
Kingston
{devin.pereira, chris.molloy, s.acharya<EMAIL_ADDRESS>

###### Abstract

It is becoming increasingly important in the software industry, especially
with the growing complexity of software ecosystems and the emphasis on
security and compliance for manufacturers to inventory software used on their
systems. A Software-Bill-of-Materials (SBOM) is a comprehensive inventory
detailing a software application’s components and dependencies. Current
approaches rely on case-based reasoning to inconsistently identify the
software components embedded in binary files. We propose a different route, an
automated method for generating SBOMs to prevent disastrous supply-chain
attacks. Remaining on the topic of static code analysis, we interpret this
problem as a semantic similarity task wherein a transformer model can be
trained to relate a product name to corresponding version strings. Our test
results are compelling, demonstrating the model’s strong performance in the
zero-shot classification task, further demonstrating the potential for use in
a real-world cybersecurity context.

##  1 Introduction

Throughout the past decade, major security breaches have been caused by
malicious code embedded in firmware, a type of software that directly
interacts with hardware. Attacks on firmware can be particularly revealing and
destructive, as firmware has high privileges over other aspects of computer
systems. Recently, the devastating effects of exploiting firmware
vulnerabilities were evidenced by the SolarWinds supply chain attack [1], an
attack that compromised thousands of organizations around the world, including
the United States government.

A supply chain attack constitutes a security breach where a malicious entity
exploits vulnerabilities within third-party software to gain unauthorized
access to the targeted system rather than attempting a direct breach of the
target system itself. Supply chains associated with contemporary software
systems are evolving into more intricate structures, contributing to increased
opacity and making them susceptible to such attacks. It follows that it has
become extremely difficult to identify which components are present in a
particular piece of software [2]. The problem finds its source in the software
engineering practice of code reuse. Since this custom is unlikely to be
abandoned, an alternative approach to prevent supply chain attacks is
required.

In 2018, the National Telecommunications and Information Administration of the
United States Department of Commerce (NTIA) formed a working group that
proposed the Software-Bill-of-Materials (SBOM). Concisely, an SBOM provides
details about all the components present in a given piece of software. SBOMs
detail information such as the software supplier, the component names and
versions present, as well as the relationships between these components.

However, it is important to note that SBOMs do not guarantee security
throughout the supply chain. An SBOM is not applicable against zero-day
exploits, nor can it be used to flag a security flaw before it is reported to
a vulnerability database [3]. Nevertheless, the simplest approach to derive an
SBOM is to use the relevant package manager to list a software’s dependencies.
Though simple and highly effective, it requires knowledge of the package
manager used for installation. Evidently, the technique does not work for
software installed without a package manager.

The alternative procedure of deriving an SBOM from a software’s source code is
equally reliable. However, the main limitation of this approach is that source
code is not guaranteed to be available. In general, it is near impossible to
locate source code for a specific binary, particularly in the case of
firmware, which often is preinstalled on systems [3]. There is a salient need
for a tool capable of producing SBOMs using only compiled binaries. Such a
tool could guarantee the integrity of vendor-generated SBOMs and protect
against tampering by adversaries, i.e., a man-in-the-middle attack in which a
software is maliciously modified, but its SBOM deceitfully remains the same.

Henceforth, we refer to SBOM generation tools as utilities for deriving SBOMs
from binaries. Tools like these exist but have the drawback of relying on
case-based reasoning. In fact, many of them require predetermined unique
patterns for matching strings to a specific set of software products.

To remedy this issue at scale, we elect to implement a machine-learning
solution for matching version strings to software products. We demonstrate
that this problem can be solved using modern natural language processing
techniques. Our transformer model is tasked with determining the semantic
similarity between a software product name and the version strings embedded in
software binaries. The most similar software product to the version string is
then selected as the network’s prediction. We intend that these predictions be
used as input for the automated generation of SBOMs and to query vulnerability
databases if there are CVEs corresponding to that particular product. We show
our model’s ability for zero-shot inference by testing its detection accuracy
on multiple software versions not encountered in training. Our contributions
can be summarized as follows:

  * •

We propose the first transformer model optimized for the SBOM generation task.
As opposed to a naive pattern-matching operation, identifying software modules
is formulated as a semantic-similarity-based classification task.

  * •

By way of zero-shot inference experiments, we demonstrate that our model,
despite being trained on a few classes, can be applied successfully with only
a small degradation in performance when compared to a state-of-the-art model
trained on all available classes. The outcome underscores our method as being
effective in practical cybersecurity contexts.

The paper’s structure is as follows. The _Related Works_ section highlights a
series of studies from which we draw inspiration. The _Similarity Learning for
Version Detection_ section contains the preparation of our dataset, a
description of the data employed, and a illustration of our model. In
_Experimental Setup and Results_ section, we explain our experiments and
describe our interpretation of the results. Finally, the paper concludes and
provides potential scopes for future works in _Conclusion and Future works_
section.

##  2 Related Works

In this section, we review the current tools for generating SBOMs. We appraise
studies wherein transformer models have been effectively employed for pattern
detection. Finally, we present approaches to classification tasks which
involve zero-shot learning.

###  2.1 SBOM Generation Tools

Intel’s open source CVE-bin-tool is a widely used software analysis program
which employs a binary scanner to determine which software versions are
embedded in a binary file and subsequently maps these versions to known
vulnerabilities in the National Vulnerability Database [4]. Doan and Jung
employ CVE-bin-tool to develop DAVS–A Dockerfile analysis-based vulnerability
scanning framework [5].

Their approach aims to create a methodology for vulnerability detection that
is not restricted by the ability of package managers to extract information.
After determining potentially vulnerable files through preliminary dockerfile
analysis, they make use of CVE-bin-tool to detect software versions, linking
them to vulnerabilities. Reinhold et al. compare the results produced by
different versions of CVE-bin-tool on 660 binary files [6]. They observe that
only $1$ out of the $660$ binaries (excluding null results) produces an output
consistent across the different versions the tool. This outcome raises
additional doubts about the reliability of the case-based approach and
underscores the necessity for a new paradigm.

Figure 1: Binaries obtained from different sources are passed through
extraction scripts. Next, strings are generated from the resultant ELF files.
Finally, these strings are funneled through a regular-expression based
filtering script. The final data points can be observed as a tuple with four
different fields.

###  2.2 Transformer Models

Very little research addresses the problem of determining the similarity
between pseudo-English words. Such is the nature of the task in comparing a
collection of version strings derived from binary files to software names.
However, the current state-of-the-art class of models has been used
extensively for classification tasks, which we consider to be a related
problem type to semantic similarity. Shaheen et al. demonstrate the efficacy
of using transformer models in text classification [7]. To achieve state-of-
the-art results in Large Multi-Label Text Classification (LMTC), they compare
different models alongside various strategies such as generative pretraining,
gradual unfreezing, and discriminative learning rates. Across the different
datasets, BERT (Bidirectional Encoder Representations from Transformers)
variants were observed to yield the best results. Interestingly, the
researchers note that multilingual BERT trained only in English data yields
results on par with the others when presented with texts from other languages.
This indicates BERT’s zero-shot learning capabilities.

BERT is the current state-of-the-art embedding model for natural language
introduced by Devlin et al. [8]. Their novel bidirectional attention model
allows for learning information from a text both left to right and right to
left. In order to develop a deep bidirectional representation, the researchers
mask some percentage of the input tokens at random during training and then
predict the masked tokens. This procedure is known as masked language modeling
(MLM). Prior left-to-right-only approaches restricted models to unidirectional
learning. BERT has shown to be seminal as it can be used as a foundation for
other models. Fine-tuning can be achieved with as little as one additional
output layer to create state-of-the-art models for specialized natural
language tasks.

In the year following the release of BERT, Nils Reimers and Iryna Gurevych
proposed a method to increase the efficiency of BERT in a sentence similarity
task. [9]. The motivation behind their improvement relates to a shortcoming of
BERT: its cross-encoder setup is unsuitable for a variety of pair regression
tasks due to the excessive number of possible output combinations it allows.
Instead, the researchers develop a Siamese network architecture which enables
the derivation of fixed-sized vectors from input sentences. These fixed-size
vectors can be viewed as semantically meaningful sentence embeddings. They
assert that, in a task to determine the similarity between $10,000$ sentences,
the necessary computation time is reduced from $65$ hours with BERT, to around
$5$ seconds with S-BERT. In brief, their improved model drastically reduces
the computational overhead, thereby improving the time taken to determine the
most semantically similar pair of sentences. Simultaneously, their method
almost perfectly maintains the near stellar accuracy of BERT.

###  2.3 Zero-shot Learning

The formalization of zero-shot learning is attributed to Palatucci et al.
These researchers introduce a so-called semantic output code classifier, which
infers a knowledge base of semantic properties from samples in a training set.
They demonstrate that their classifier can accurately predict outputs from
novel classes, emphasizing the relevance of the task when outputs have the
potential to take on a large number of different values, as well as when the
cost of obtaining labeled data is high. The model used in their experiments is
made to predict words using functional magnetic resonance images from human
participants. They justify the necessity of their novel procedure by stating
that determining neural training images for every English word in existence is
intractable. Over two semantic knowledge bases, human218 and corpus5000, their
method obtains mean accuracies of $80.9\%$ and $69.7\%$, respectively. These
results were shown to be powerful, despite originating from the brain waves of
only nine participants. From its inception, the concept of zero-shot learning
was linked to classification tasks with a semantic character. [10]

Ziming Zhang and Venkatesh Saligrama describe the zero-shot learning problem
from an alternative perspective. They view target data instances as arising
from observed instances, attempting to express source and target data as a
mixture of observed class proportions. Their idea is, if the mixture
proportion from the target domain is similar to that from the source domain,
the target and source necessarily originate from the same class. Consequently,
they formulate the problem as learning source and target domain embedding
functions using observed class data. These functions are designed to map
arbitrary source and target domain data into mixture proportions of observed
classes. Their method involving semantic similarity embedding functions
improves the state-of-the-art accuracy CIFAR-10 to $88.3\%$ amongst other
benchmarks. [11]

##  3 Similarity Learning for Version Detection

###  3.1 Data Preparation

We require a dataset of varying software products and version strings for
training our network. We collect product and version information from Portable
Executable (PE) and Executable and Linkable Format (ELF) file type binaries.
The PE format used by Windows specifies the structure of executable files
(.exe) and Dynamic Link Libraries (DLLs), which are linkable software
packages. Though there are many other file types which fall under the PE
umbrella, we collect only these two in our dataset. Note that these are
designed for many architectures e.g. x86, arm, amd64, etc. Analogous to PE but
conceived for Unix systems, the ELF format is commonly used for executables,
object code, shared libraries, and more.

We obtain our files from two sources: conan.io and debian.org. In the case of
conan.io, we query the _conan-center_ to download files from over $1500$
different libraries. We select only the PE and ELF files from the downloaded
directory. This list of files is compared to the reference file paths in the
cache to avoid duplication, renamed, and finally placed in the dataset. The
procedure is nearly identical for the debian packages.

Once the EFL files originating from all sources are gathered, they are
transformed into strings, simply using the Unix _strings_ command. Within
these strings, we consider only the strings that match a general regular
expression or version string pattern. From the metadata in the repository, we
obtain a package name that contains the version of the software. This version
number is used as a ground truth to label each version string. The product is
simply the name from the conan or debian package catalogue.

###  3.2 Data Description

Though SBOMs contain many other fields, we deem most of these irrelevant to
our research. The SPDX format, the standard structure for SBOMs, was
originally conceived to manage product licensing information [12]. Hence, many
of these fields are not pertinent to vulnerability analysis which we consider
to be the primary purpose of SBOMs in our study. For example, relationships
between components are not recorded, our method only identifies the components
which are present in the piece of software.

There are other fields which our approach does not directly cover.
Manufacturers can be easily extrapolated from the product name. As mentioned
above, version numbers, which are firmly required for SBOM generation and CVE
identification, are used to label the version strings and thus are not
subsequently determined by the machine learning solution.

In summary, each data point in our final processed dataset, illustrated in
Figure 1, contains a product, a package name and a list of valid version
strings. Finally, having already correlated each valid version string to the
product, we decorrelate those version strings with the other unrelated
products to optimize future classification.

###  3.3 Model

Figure 2: The product and one version string from the list is taken as a data
point which serves as input to the S-BERT model. S-BERT produces embeddings
which are pooled into two vectors, u and v. The cosine similarity is
calculated with these two vectors and contrasted with the predefined
correlation label to obtain the final classification

In BERT and S-BERT, inputs are parsed according to the WordPiece tokenization
algorithm. We will now describe the WordPiece tokenization algorithm [13]. As
an initial step, a WordPiece model must be generated through the training
procedure. It can be explained as follows: the model is initialized with
wordpieces as word units or simple characters. Given a training corpus with
stopwords filtered, the model does a pairwise combination of wordpieces to
increase the likelihood of them being present in the training corpus. This
process is continued until the minimum number of wordpieces for the corpus is
constructed. Alternatively, training may be made to stop once a predefined
number of wordpieces is attained, in which case the remainder will be out-of-
vocabulary.

Prior to training, word separators are added such that the original sequence
of words can be recovered from the wordpiece sequence produced by the model.
BERT uses two special tokens in its version of the WordPiece algorithm. The
special classification token [CLS], whose final hidden state is used as the
aggregate sequence representation in the classification task, and the [SEP],
which delineates the two inputs. As well, many rare characters are not
represented to minimize inefficiencies. Instead, these are replaced by an
unknown placeholder character ([UNK]). An additional constraint is that input
sentences longer than $256$ word pieces are truncated. Notably, this approach
remains unaffected by word or sentence semantics.

In the context of our model, the WordPiece tokenizer can be described as a
mapping from $\mathbb{Z}^{U*l}\rightarrow\mathbb{Z}^{W*l}$ where $U$ is the
number of unicode representations, $l=256$ is the maximum input sequence
length and $W=30,000$ is the number of wordpieces. These vectorized wordpieces
are then combined with positional embeddings and binary segment embeddings to
indicate provenance from the first or second sentence. The procedure can be
observed as $\mathbb{Z}^{W*l}\rightarrow\mathbb{R}^{H*l}$ where $H=768$ is the
hidden size or the size of the embeddings for each wordpiece. Subsequently,
these BERT embeddings are transformed according to the mean-pooling operation
$\mathbb{R}^{H*l}\rightarrow\mathbb{R}^{e}$ where $e=384$ is a dense vector
space representing the fixed-size of the S-BERT embedding vector which
captures the entire input sequence. These embeddings are used as inputs to an
objective function $S(x)$ from $\mathbb{R}^{e}\rightarrow 0<\mathbb{R}<1$. The
resulting real number is the probability that a given product name corresponds
to the queried version string.

Table 1: All $20$ classes present in the training set. The training set contains $4,000$ samples from each. agda | ghc | gcc-xtensa-lx106 | fricas | gcc-13
---|---|---|---|---
grass | haskell-opengl | haxml | gcc-12-cross-mipsen | grub2
pandoc | haskell-gtk | haskell-graphviz | gcc-riscv64-unknown-elf | haskell-gtk3
darcs | metaeuk | uuagc | gcc-10-cross | propellor

To compute the sentence similarity score from two inputs, we employ an S-BERT
implementation from the Sentence Transformers package, specifically the _all-
MiniLM-L6-v2_ model. It is structured with three components as highlighted
above: 1) the BERT model, 2) the pooling layer, 3) the similarity function.
Our model employs BERTBASE, which has $12$ standard multi-layer bidirectional
Transformer layers, $12$ self-attention heads, and a hidden size of $768$
comprising 110M total parameters. With up to $256$ outputs of hidden size
emanating from this network, the mean-pooling layer condenses the information
into a fixed-size embedding vector. We calculate similarity scores using these
embeddings as input. Finally, the cross-entropy loss is computed on the
cosine-similarity score and the predicted label.

$cosine\\_sim(u,v)=\frac{u\cdot v}{\|u\|\cdot\|v\|}$ (1)

In a deployed setting, we expect this tool to be used by a stakeholder who is
likely to be a security analyst or a systems administrator. In the first
phase, the user must gather as many software binaries as is available to their
organization. They must then process the files in their repository following
our pipeline to obtain products and a corresponding list of version strings.
Importantly, product names must be made consistent for optimal performance.
Following our methodology, the model can be trained and tested. At this point,
it is ready for practical use. One simple use case involves an organization
wanting to install new software in its system. This tool can be relied upon to
decide whether the software can be whitelisted. If, after analysis, the model
determines that none of the version strings in the software file correspond to
a product linked with CVEs, then the software is safe to install. Optionally,
the file can be added to the dataset for future training.

##  4 Experimental Setup and Results

We obtain the raw data, a collection of over 100GB of binaries, from various
online repositories. From the raw data, we curated a dataset of $400,000$
version pattern samples sourced from a wide range of software binaries.

###  4.1 Experiments

To validate the proposed methodology, we conducted three experiments. The
first experiment serves to verify the usefulness of semantic similarity for
making predictions about our data. Also, it is designed to guide our selection
of the similarity metric to use for later experiments. The second experiment
compares the model’s efficacy across two different data sampling techniques.
It contrasts a perfect simulation run in the lab with a demonstration of how
the tool might be employed in practice. We use a training set with
significantly less sample diversity to account for the low number of malware
samples available to the average user in comparison to the abundance of
software ecosystems. Through the third experiment, we determine the optimal
training length for our model when used in a practical setting.

We limit the size of the experimental dataset to $100,000$ samples in all
experiments. We deem this number to promote both sufficient sample diversity
and training efficiency while maintaining the quality of the results, which
can be achieved on the entire dataset. Following a similar line of thinking,
we consistently split the data into training ($80\%$) and testing ($20\%$)
sets.

Table 2: An extract of the classes in the testing set. Class Name | N Samples | Density
---|---|---
systemd | 86 | 0.430%
libreoffice | 72 | 0.360%
espresso | 39 | 0.195%
medusa | 13 | 0.065%
js-of-ocaml | 3 | 0.015%
scrypt | 1 | 0.005%
xmppc | 1 | 0.005%
Table 3: The performance of the model when comparing two different similarity metrics in the zero-shot experiment. Similarity Metric | Accuracy | Precision | Recall |
---|---|---|---|---
Cosine | 0.9290 | 0.9447 | 0.9126 |
Dot Product | 0.9267 | 0.9452 | 0.90687 |
Table 4: The results compared by inference type. Fully trained in the case of experiment one where the model is trained on all classes. In the zero-shot experiment, our model is trained on only $20$ classes. Inference Type | Accuracy | AUC | Precision | Recall | F-1 Score
---|---|---|---|---|---
Fully-Trained | 0.9290 | 0.9810 | 0.9447 | 0.9126 | 0.9284
Zero-Shot | 0.8518 | 0.9016 | 0.9512 | 0.7410 | 0.8330
Table 5: The zero-shot results in different training runs over increasing epochs demonstrate overfitting. Epoch | Accuracy | AUC | Precision | Recall | F-1 Score
---|---|---|---|---|---
1 | 0.8518 | 0.9016 | 0.9512 | 0.7410 | 0.8330
2 | 0.8225 | 0.8783 | 0.9967 | 0.6463 | 0.7841
5 | 0.7383 | 0.7733 | 0.9777 | 0.4866 | 0.6498
10 | 0.5579 | 0.6267 | 0.6901 | 0.2068 | 0.3182

In the first experiment, we randomly select and assign samples from all
classes to the two subsets, with the possibility of both sets containing
samples from the same class. Then as described above, our S-BERT variant
processes this data with a batch size of $512$, moulding embeddings as part of
the training process. These embeddings are combined using either cosine or dot
product similarity, producing a semantic similarity score, forming the basis
for our comparison metrics displayed in Table 3.

The second experiment makes use of two different data selection procedures. In
the fully-trained approach, the data is split with samples randomly selected
from all classes in the same way as the first experiment. The zero-shot method
begins by grouping all samples by class or library. These classes are sorted
by the number of samples in descending order to facilitate the filling of the
training set.

To match the predetermined size of the training set, $4,000$ samples from each
of the $20$ largest classes are drawn and exhibited in Table 1. To the testing
set, we randomly allocate samples from the remaining classes in the initial
dataset. A subset of the constituents to the testing set can be observed in
Table 2. Again the model produces similarity scores, but this time only using
dot product similarity. We amalgamate these scores to compute a variety of
metrics qualifying the effectiveness of our zero-shot method. These metrics
can be seen in Table 4.

In the third experiment, following the same zero-shot procedure as in the
second experiment, we conducted four zero-shot trials over an increasing
number of epochs. The results can be observed in Table 5.

The first experiment demonstrates that results are largely insensitive to the
choice of similarity metric. However, as expected, the cosine similarity
metric used to train the model performs best in testing. It outperforms dot
product similarity in both accuracy and, notably, recall. Due to minor
variations in data selection, it could be observed in a similar experiment
that the accuracy of dot product similarity slightly outperforms that of
cosine similarity. In this context of cybersecurity, where prevention is
paramount, we would still advocate choosing cosine similarity if it captures
more positive samples, thereby giving rise to a higher recall score.

$\sum_{i=0}^{n=384}u_{i}\cdot v_{i}=u_{0}\cdot v_{0}+u_{1}\cdot
v_{1}+...+u_{n-1}\cdot v_{n-1}$ (2) $u\cdot v=|u|\cdot|v|\cdot cos(\alpha)$
(3)

Results from the second experiment indicate that the fully-trained model
performs well, as is our expectation when resented with training and testing
data sampled from the same classes. But importantly, we find the model’s
performance in the zero-shot experiment to be remarkable. Despite training on
only $20$ classes in the zero-shot task, the model’s accuracy differs by a
little over $7\%$ in comparison to the first experiment. Evidently, the recall
score suffers from the poverty in sample diversity, leading to the model’s
inability to recognize as many positive samples. Upon closer inspection, there
seems to be a particular maximum number of samples per class or a specific
number of classes that is optimal for the zero-shot performance. We determined
a local maximum for performance at $4,000$ samples and $20$ classes.

The outcome of the third experiment concerning the number of epochs used for
training is quite simple. The original S-BERT model, itself a fine-tuned
version of BERT, was trained over a single epoch. Hence, it is logical that
the best results would be obtained by fine-tuning S-BERT over one epoch. In
this instance, anything more causes a degradation in performance due to
overfitting.

We demonstrate through our experiments that our approach is a solution to the
problem of inconsistent results highlighted by Reinhold et al. Provided that
data is processed with care, any number of different version strings when
correctly classified, will consistently match to the same product.
Furthermore, existing tools, such as cve-bin-tool and EMBER, only support
around $300$ packages, whereas we were able to apply the model to the whole
debian repository, which contains over $40,000$ packages. From a practical
perspective, this paper describes a method for determining software products.
Though it uses version numbers, it does not directly address the problem of
identifying them within these products. Nonetheless, version numbers are an
important aspect of SBOMs. In fact, they are critical for identifying
vulnerable components, as vulnerability databases require version numbers to
accurately relate a software component to a vulnerability. Be that as it may,
we deem the problem of version number identification to be a trivial operation
involving regular expressions. Therefore we have not considered that problem
in this paper.

Reflecting further upon the second experiment, some of the training classes
may appear to have version strings with similar structures, for example,
haskell-gtk and haskell-gtk3. The relationship between the semantic similarity
of version strings from different products in the training set, and zero-shot
test performance should be a primary focus of future investigation. At first
glance, the solution to the problem might be formulated as a fully connected
graph, where the vertices are a subset of the products available, curated for
the training set such that the edges are the similarity scores producing a
minimum sum for the dataset. Such analysis might be supplemented with a closer
look at test results which naturally reveal differences in performance by
class. Observing the structure of version strings in the classes which yield
low recall scores may also guide the selection of classes in forming an
optimal training set.

##  5 Conclusion and Future Works

The purpose of this study is to address the critical issue of supply-chain
attacks, whose notoriety was cemented by the Solarwinds supply-chain attack.
Recognizing the critical nature of SBOMs in ensuring supply chain security, we
brought forward a novel solution that leverages machine learning, specifically
the transformer model S-BERT, to automate the generation of SBOMs from
compiled binaries. Existing SBOM generation tools, which are generally reliant
on case-based reasoning, have demonstrated limitations in dealing with the
diverse and dynamic nature of modern software productions.

In an effort to address these challenges, we introduced a method which employs
S-BERT to match different version strings present in binaries to the
appropriate software product. Three experiments were conducted. The first
guided our choice of similarity metric. The second contrasted an instance of
our model trained on all classes to one trained on a reduced set to simulate
real-world scenarios with zero-shot learning. The third addressed the impact
of the number of epochs on the model’s performance, emphasizing the risk of
overfitting.

Results showcased the model’s impressive zero-shot inference capabilities as
it achieved strong results. Though, our findings highlight the importance of
sample diversity, raising the question of how class selection should occur for
forming training sets. As an open problem for subsequent research, we leave
optimizing class selection for training set composition. For a more
comprehensive approach to supply chain security, the identification of version
numbers within software products is a simple yet important endeavor to
undertake in the future. We note that this component is the only other data
point required to generate a reduced SBOM sufficient for CVE detection. In
conclusion, our research has the potential to protect large scale software
supply chains using very little data, thereby making it an extremely valuable
tool for protecting organizations and individuals from malicious actors.

## References

  * [1] Lavi Lazarovitz.  Deconstructing the solarwinds breach.  Computer Fraud & Security, 2021(6):17–19, 2021.
  * [2] Éamonn Ó Muirí.  Framing software component transparency: Establishing a common software bill of material (sbom).  NTIA, Nov, 12, 2019.
  * [3] Amas Phillips, Carsten Maple, Florian Lukavsky, Ian Pearson, Michael Richardson, Nigel Hanson, Paul Kearney, and Robert Dobson.  Software bills of materials for iot and ot devices.  IoT Security Foundation, 2023.
  * [4] Harold Booth, Doug Rike, and Gregory Witte.  The national vulnerability database (nvd): Overview, 2013-12-18 2013.
  * [5] Thien-Phuc Doan and Souhwan Jung.  Davs: Dockerfile analysis for container image vulnerability scanning.  Computers, Materials & Continua, 72(1), 2022.
  * [6] Ann Marie Reinhold, Travis Weber, Colleen Lemak, Derek Reimanis, and Clemente Izurieta.  New version, new answer: Investigating cybersecurity static-analysis tool findings.  In 2023 IEEE International Conference on Cyber Security and Resilience (CSR), pages 28–35. IEEE, 2023.
  * [7] Zein Shaheen, Gerhard Wohlgenannt, and Erwin Filtz.  Large scale legal text classification using transformer models.  arXiv preprint arXiv:2010.12871, 2020.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [10] Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell.  Zero-shot learning with semantic output codes.  Advances in neural information processing systems, 22, 2009.
  * [11] Ziming Zhang and Venkatesh Saligrama.  Zero-shot learning via semantic similarity embedding.  In Proceedings of the IEEE international conference on computer vision, pages 4166–4174, 2015.
  * [12] NTIA, 2021.
  * [13] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144, 2016.


# Learning to Look Inside: Augmenting Token-Based Encoders with Character-
Level Information

Yuval Pinter
School of Interactive Computing
Georgia Institute of Technology
Atlanta, GA, USA
<EMAIL_ADDRESS>
&Amanda Stent
Bloomberg
New York, NY, USA
<EMAIL_ADDRESS>
Mark Dredze
Bloomberg
Department of Computer Science
Johns Hopkins University
<EMAIL_ADDRESS>
&Jacob Eisenstein
Google Research
<EMAIL_ADDRESS>
Work done as an intern at Bloomberg LP and as a Bloomberg PhD Data Science
Fellow.

###### Abstract

Commonly-used transformer language models depend on a tokenization schema
which sets an unchangeable subword vocabulary prior to pre-training, destined
to be applied to all downstream tasks regardless of domain shift, novel word
formations, or other sources of vocabulary mismatch. Recent work has shown
that “token-free” models can be trained directly on characters or bytes, but
training these models from scratch requires substantial computational
resources, and this implies discarding the many domain-specific models that
were trained on tokens. In this paper, we present XRayEmb, a method for
retrofitting existing token-based models with character-level information.
XRayEmb is composed of a character-level “encoder” that computes vector
representations of character sequences, and a generative component that
decodes from the internal representation to a character sequence. We show that
incorporating XRayEmb’s learned vectors into sequences of pre-trained token
embeddings helps performance on both autoregressive and masked pre-trained
transformer architectures and on both sequence-level and sequence tagging
tasks, particularly on non-standard English text.

##  1 Introduction

The use of subword representations in NLP applications has become the default
choice in recent years, most notably within the framework of large pre-trained
language models (LPLMs) where the LM objective requires a vocabulary that is
computationally feasible for a softmax operation, typically interpreted as no
larger than $10^{5}$ types, while also exhaustive, such that it allows
production of arbitrary space-delimited tokens in order to mitigate the out-
of-vocabulary (OOV) problem. Informative subword vocabularies are pre-computed
by methods such as byte-pair encoding (BPE; Sennrich et al., 2016) and its
variant WordPiece used in BERT Devlin et al. (2019), or Unigram LM Kudo
(2018). Such vocabularies normally make use of explicit signalling for either
word-medial or word-boundary tokens, so that recompiling the original space-
delimited words is a deterministic process.111In this paper, we refer to
space-delimited sections of text as words even when they are not linguistic
words per se, in order to distinguish them from the model-centric tokens which
are represented and operated over by LPLMs.

#simplyleopard #4thofjuly #showmeyourmumu @ Town of Breckenridge, Colorado
---
# simply ##le ##opa ##rd # 4th ##of ##ju ##ly # show ##me ##you ##rm ##um ##u
@ town of br ##eck ##en ##ridge , colorado
Table 1: A tweet (top) and its WordPiece tokenization (bert-base-uncased,
bottom).

However, processes for constructing subword vocabularies have high variance:
perturbing the dataset used to train them leads to significant changes in the
resulting vocabulary. In a preliminary experiment, we sampled two sets of 1
million sentences from the same Wikipedia dump and trained a Unigram LM model
of 32,000 tokens on both. The resulting vocabularies presented a discrepancy
of 27% (8,620 unshared tokens). Similarly, Lazaridou et al. (2021) found that
collecting corpus data from different timestamps of the same source shifts the
vocabulary towards terms used more frequently over different times.
Nevertheless, this creation procedure is irreversible: once a subword
vocabulary has been set, the LPLM is tied down to it regardless of data
encountered in subsequent LM training and fine-tuning.

$X$Myhovercraftisfullofeels.$\tau$myhover##craft[MASK]full[MASK]eels.$\pi^{t}$
✓ $\times$ ✓ ✓ ✓ ✓ ✓ $\mathbf{E}$XR-Ench o v e r c r a f t /w$M$$\pi^{g}$
$\times$ ✓ $\times$ PredofXR-Deci s /we e l s /w Figure 1: XRayEmb integration
into a masked language model. $\pi^{t}$ is set to send all multi-WordPiece
words into XR-Enc; $\pi^{g}$ is set to generate every third word using XR-Dec.
/w is a reserved end-of-word character.

When shifting between domains, topics, and registers, this property produces
undesired consequences Blitzer et al. (2007). One effect of training or fine-
tuning an LPLM over a dataset extracted from a domain other than the one on
which its subword vocabulary was created is the formation of long input
sequences with different words receiving markedly different treatment. For
example, in the tweet shown in Table 1, taken from a task where an emoji is to
be matched to a tweet (see §3), the intended emoji is the US flag; but the
excessive number of WordPiece tokens, and the breaking down of the most
informative word #4thofjuly into four unrecognizable segments, including the
splitting of July due to its mid-word location, causes a classifier trained
over a fine-tuned BERT model to falsely predict the red heart emoji. In
sequence labeling tasks like named entity recognition or morphological
tagging, the breaking down of sequence-atomic words into tokens presents a
nontrivial decision point as to how multi-token words’ output representations
are to be used for predicting labels Ács et al. (2021). In languages with non-
concatenative morphology such as Hebrew, the contiguous nature of subword
tokens has been shown to render the representations of the transformer stack
all but useless Klein and Tsarfaty (2020).

In this paper we present XRayEmb, a system which provides LPLMs the option to
represent novel and rare words with a single vector composed over their
character sequence, while retaining the knowledge learned for well-recognized
word tokens during pre-training. Our method bridges the gap between the two
representation modules via an additional pre-training sequence where the
language modeling objective is supplemented with training a character-level
encoding component which provides vector inputs to the contextualized model
apparatus, as well as a generative component which learns to output target
words from their contextualized representations. We show how these lightweight
auxiliary components can be trained so that they regularize each other,
resulting in a character-to-vector model that approximates representations
well for single- and multi-token words, and in a word-generation model which
produces well-formed sequences approximating English from arbitrary points in
vector space. We examine the performance of various LPLMs trained with a
XRayEmb module over text from both user-generated and edited sources on both
sequence classification and sequence labeling tasks, finding that our system’s
strength lies mostly in the latter. We then analyze the effect of individual
components and decision points in our system, such as our choice of corpus for
the second pre-training phase and our regularization regime.222We will release
our code and model checkpoints.

##  2 XRayEmb

We focus on character-informed representations for LPLMs which use the
transformer architecture Vaswani et al. (2017), such as BERT Devlin et al.
(2019), RoBERTa Liu et al. (2019), and GPT Radford et al. (2018). A
Transformer LPLM is composed around a core module $M$ parameterized as multi-
head self-attention layers, which accepts a sequence
$\\{\bm{e}_{1},\dots,\bm{e}_{l}\\}$ of embedding vectors corresponding to a
list of token indices $\\{t_{1},\dots,t_{l}\\}$ from a vocabulary
$\mathcal{V}$, and outputs a sequence of contextualized vectors
$\\{\bm{h}_{1},\dots,\bm{h}_{l}\\}$, of which a subset
$\\{\bm{h}_{i}\\}_{i\in\mathcal{I}}$ correspond to masked tokens in positions
$\mathcal{I}\subseteq[l]$ selected by a boolean masking operator $m$.333In the
case of an autoregressive LPLM like GPT, $\mathcal{I}=[l]$ but each vector
$\bm{h}_{i}$ is only dependent on the tokens preceding its position,
$\\{t_{1},\dots,t_{i-1}\\}$, and token-wise prediction is performed for each
position assuming no knowledge of future tokens. The token index list supplied
to $M$ is the output of a tokenization function $\tau$ operating on a sequence
$X$ followed by lookup in an embedding table $\bm{E}$, while its output
vectors $\\{\bm{h}_{1},\dots,\bm{h}_{l}\\}$ serve as input to a prediction
module Pred, which outputs a distribution $D_{i}$ over $\mathcal{V}$ for each
masked position $i$. All embeddings $\\{\bm{e}_{i}\\},\\{\bm{h}_{i}\\}$ live
in a shared space $\mathds{R}^{d}$. Together, the components described so far
operate in the following manner:

$\\{D_{i}(\mathcal{V})\\}_{i\in\\{\mathcal{I}\\}}=\text{{Pred}{}}\left(\text{{$M$}{}}\left(\bm{E}\left[\tau(X)\right]\right)\right),$
(1)

where square bracketing denotes elementwise table lookup.

In the considered LPLM architectures, the input $X$, which is atomically made
up of a sequence of characters $\\{c_{1},\dots,c_{n}\\}\in\Sigma^{n}$, is
broken down by $\tau$ to provide the tokenization of length $l\leq n$, and the
prediction/generation operator Pred accepts the contextualized outputs
$\\{\bm{h}_{1},\dots,\bm{h}_{l}\\}$ and implements prediction by means of a
softmax distribution which is based on scores obtained via dot-product against
an output embedding table, which is usually the same as $\bm{E}$.

The XRayEmb model makes no adjustments to $M$ itself, and only offers
conditional replacements for $\tau$ and Pred. An encoding policy $\pi^{t}$
selects a subset of tokens from the original sequence
$\mathcal{J}^{t}\subseteq[l]$ to be represented by XR-Enc instead of
$\bm{E}\circ\tau$. XR-Enc has access to the part of the character sequence $X$
which underly the tokens selected by $\pi^{t}$, and produces input embeddings
directly from the character level. These alternate embeddings must agree in
dimension with those of each $\bm{e}$, but a single embedding may be used to
replace multiple base tokens (usually when all tokens corresponding to a
single out-of-$\bm{E}$ word are replaced), resulting in a shorter input
sequence for $M$. A separate generation policy $\pi^{g}$ selects a subset of
the original sequence $\mathcal{J}^{g}\subseteq[l]$ to be generated from the
output vectors $\bm{h}$ by XR-Dec instead of Pred. A high-level schematic
depicting this framework is presented in Figure 1.

Original | He was emphatically a modern gentleman, of scrupulous courtesy, sportive gaiety,
---|---
Word pieces | He was em #pha #tically a modern gentleman , of s #c #rup #ulous courtesy , sport #ive g #ai #ety ,
Random-20% | He was [TOK] a [TOK] gentleman , of s #c #rup #ulous [TOK] , sport #ive g #ai #ety ,
All-multi | He was [TOK] a modern gentleman , of [TOK] courtesy , [TOK] [TOK] ,
Suffixes | He was [TOK] a modern gentleman , of [TOK] courtesy , sport #ive [TOK] ,
Table 2: Example of token input selection by different policies, where [TOK]
signifies a word to be replaced by its character-based representation from XR-
Enc. Sentence fragment taken from the MS-MARCO QA dataset (see §3) and
tokenized using BERT-cased (## replaced with # to fit paper width).

The specific policies in a given application may be defined based on the
model’s use case. For example, in text classification no generation is
required, and so $\pi^{g}$ will return $\emptyset$ for all sequences;
$\pi^{t}$ can be tuned for a task based on known features of the base model
(BERT/GPT etc.) and of the domain text, some examples including: tokens
corresponding to all words that are not in a pre-determined vocabulary; all
words in the sequence; all words assigned more than one token by the base
tokenizer $\tau$; a random sample of words in the sequence; or all words
including characters that are not lowercase English characters. One particular
policy we hypothesize could be useful is one that affords the tokenizer slack
in detecting a single simple derivational or inflectional suffix: all words
which are single-token or whose second-and-final token is in the list Suffixes
are left for $\bm{E}\circ\tau$; the rest are represented using XR-
Enc.444Suffixes, compiled by manually examining a list of most common second-
and-final tokens in a large corpus under GPT-2’s tokenization, = $\\{$_s_ ,
_ed_ , _es_ , _ing_ , _ly_ , _al_ , _ally_ , _’m_ , _’re_ , _’ve_ , _y_ ,
_ive_ , _er_ , _’t_ , _’ll_ , _an_ , _ers_ $\\}$. Similar policies can be
hand-crafted for different languages, based on learner-level knowledge of the
language and minimal preliminary analysis of sample tokenizations. Our results
show they are not strictly necessary. Different policies may be applied in
training settings as well, for example in order to familiarize the heavily-
parametrized $M$ with the inputs from XR-Enc. Three example policies are
illustrated in Table 2.

###  2.1 Second Pre-training

A typical LPLM is initiated through a computationally-intensive pre-training
step, iterating over a large corpus in batches of sequences and
backpropagating a cross-entropy loss calculated over the prediction layer’s
output into all of its components, through Pred to $M$ to $\bm{E}$. In order
to train XRayEmb, we introduce a second pre-training step we term 2pt, where
the LPLM continues to update its parameters for a (possibly different) corpus,
but is supplemented with the XRayEmb elements in order to acclimatize the $M$
components to outputs from XR-Enc.555Within the taxonomy of phases between
pre-training and task training Ruder (2021), this is closest to Adaptive Fine-
Tuning. However, due to our added modules we opt to assign it a new name. In
addition, XR-Enc is also trained through a lower-level objective requiring it
to approximate the outputs of $\bm{E}\circ\tau$ which it is replacing, and XR-
Dec is trained to sequentially produce the correct character sequence from
$M$’s outputs. Together, a batch of text sequences in a 2pt step produces the
following loss elements which are backpropagated into the unified model:

  * •

A language modeling loss from the softmax operation over masked tokens,
updating $M$’s and $\bm{E}$’s parameters, as well as XR-Enc’s for tokens
selected by a usage policy $\pi^{t}_{(u)}$ to be used in $M$’s input;

  * •

An embedding loss for the XR-Enc component, computed against
$\bm{E}\circ\tau$’s token embeddings, over a set selected by a policy
$\pi^{t}_{(l)}$ which may or may not equal $\pi^{t}_{(u)}$. This loss can be
computed, e.g., as the euclidean distance between the output and the target.
When the target corresponds to multiple token counts, an aggregation pooling
function
$\operatorname*{agg}:\mathds{R}^{d\times\mathds{N}^{+}}\rightarrow\mathds{R}^{d}$
needs to be defined over the embeddings, for example taking their dimension-
wise mean, taking the leftmost token’s embedding, or taking the dimension-wise
$\max$;

  * •

A generation loss for words generated by XR-Dec from $M$’s output vectors,
according to a $\pi^{g}$ policy. This loss is the character-level cross-
entropy for autoregressive sequence generation. Note that in order to only
generate full words, $\pi^{g}$ must align with $\pi^{t}_{(u)}$ so that no
multi-token words left as input to $M$ are also selected for generation.

As a form of regularization within the XRayEmb components, we introduce
additional training batches we call cycle dependency loops. In such a batch,
XR-Enc and XR-Dec act in succession, starting from one of the spaces they
operate in, with the goal of arriving at the same point after cycling through
both components. An E$\rightarrow$D loop thus starts at a character sequence
$\bm{c}\in{\Sigma^{*}}$, runs it through XR-Enc to obtain a vector
$\bm{e}=\textsc{XR-Enc}(\bm{c})$, and runs XR-Dec in an attempt to return to
the original sequence $\hat{\bm{c}}=\textsc{XR-Dec}(\textsc{XR-
Enc}(\bm{c}))\approx\bm{c}$. Analogously, a D$\rightarrow$E loop starts at a
vector $\tilde{\bm{e}}\in\mathds{R}^{d}$ and targets $\hat{\bm{e}}=\textsc{XR-
Enc}(\textsc{XR-Dec}(\tilde{\bm{e}}))\approx\tilde{\bm{e}}$. In this loop,
loss is only backpropagated as far as XR-Enc, since backpropagating through a
generative model’s decision component introduces discrete steps which must be
smoothed or approximated (see discussion in Peng et al. (2018)).

##  3 Tasks

To evaluate the advantages of character-sequence awareness in large-scale
transformers, we chose a diverse set of datasets which reflect unedited user-
generated language in English, as well as its interaction with edited text. We
report results on a sequence classification task (emoji prediction), a
sequence tagging task (named entity recognition, or NER) in both an in-domain
(Twitter NER) and cross-domain (emerging entities NER) setting, a sequence
ranking task based on information retrieval in a hybrid edited-unedited
textual setting (MARCO-QA ranking), and a task where a single word’s class is
predicted within a sequence (NYTWIT).

#### Emoji prediction.

The English portion of the Multilingual Emoji Prediction dataset Barbieri et
al. (2018); Ma et al. (2020a) is composed of tweets containing one of the
twenty most common emoji symbols. For the task, the emoji are stripped from
the tweet text and the system is asked to predict which one appeared in the
original tweet, allowing it to be construed as a self-annotated, fine-grained
sentiment analysis task. All tweets are identified as geographically
originating in the US. We note that there is a significant qualitative
difference between the training (+ development) set, and the test set of this
corpus, hinted at by one of the participant teams in the original task Chen et
al. (2018) but not explored.666Another team identified a seasonal shift
affecting the distribution of the Christmas tree emoji Coster et al. (2018).
The partitions were extracted based on a temporal split, with all test set
tweets post-dating all training set tweets by at least three months. More
crucially, the test set dates (May 2017 – January 2018) overlap with Twitter’s
increase of the tweet character limit from 140 to 280 characters, phased in
mostly during November 2017. As a result, test set tweets have on average
$\sim$10% more words, increasing the amount of information within and
diverging their textual feature distribution from that of the training set by
more than is customary in NLP tasks. The label distribution remains more or
less the same.

#### Twitter NER.

The Twitter NER dataset Strauss et al. (2016) is a prime example of sequence
tagging in noisy user-generated data settings. It is composed of randomly
sampled English tweets from 2016, annotated for ten different entity types,
including music artists and sports teams, thus representing topics prevalent
in social media.

#### Emerging entities recognition.

The emerging entities dataset Derczynski et al. (2017) shares most of its
training set with the same Twitter source as the Twitter NER dataset, but as a
domain-adaptation setup it includes a more ambitious evaluation fold: the
development set is extracted from YouTube comments, and the test set from
StackExchange and Reddit. The taskmasters claim that the dataset contains
mostly rare and hitherto-unseen entities, but do not provide exact statistics.

#### Community question answering.

The MS-MARCO question answering dataset (MarcoQA; Nguyen et al., 2016) was
collected by mining a commercial search engine log for user queries and asking
humans to answer them, supplying the answer writers with a set of passages
retrieved automatically from edited text by the search engine which the answer
writers then marked as selected if they helped them formulate the answer. We
recast this dataset into a selection / ranking problem, not pursued in the
original challenge: given the query and the set of possibly helpful passages,
which is the passage the answer writer selected? To this end, we filtered out
queries with more or fewer than one selected passage, and evaluated each
system based on the mean reciprocal ranks (MRR) of the true selected passages
in the rankings it produced. Due to its size, we uniformly sampled 10% of the
queries and associated passage collections from each partition in this
dataset, and ran all analyses and experiments on this new sampled dataset.

#### Novel word classification.

The NYTWIT dataset Pinter et al. (2020) includes passages in the New York
Times where a word appears which has not appeared in the publication before.
The system is tasked to classify the novel word into one of eighteen types of
novelty sources, such as inflection of known word or lexical blend. The
dataset is not partitioned into train/dev/test, and so we use the 10-fold
partition from Pinter et al. (2020) and report accuracy results aggregated
over all instances, each from a model trained on the other nine folds.

Dataset | Instances | Tokens | Types | TTR | Multitoks | Token mass
---|---|---|---|---|---|---
|  | (Space-delimited) |  |  | increase
CoNLL-2003 NER | 14,986 | 204,563 | 23,624 | .115 | 16.08% | 29.50%
Wiki1 | — | 204,564 | 28,092 | .137 | 8.64% | 12.85%
NYTWIT | 1,903 | 94,403 | 26,028 | .276 | 24.04% | 38.04%
Wiki2 | — | 94,403 | 16,903 | .179 | 9.11% | 13.75%
Twitter NER | 2,394 | 46,469 | 10,586 | .228 | 17.23% | 43.21%
Wiki3 | — | 46,509 | 10,855 | .233 | 9.02% | 13.77%
Emerging NER | 3,394 | 62,730 | 14,878 | .237 | 19.15% | 54.25%
Wiki4 | — | 62,757 | 13,392 | .213 | 9.05% | 13.74%
Emoji Prediction | 427,458 | 4,973,813 | 504,644 | .101 | 32.96% | 64.90%
Wiki5 | — | 4,973,813 | 194,240 | .039 | 9.07% | 13.58%
MarcoQA (10% sample) | 80,704 | 46,956,674 | 1,355,049 | .029 | 20.58% | 31.91%
Wiki6 | — | 46,956,703 | 818,712 | .017 | 9.48% | 14.37%
Table 3: Surface statistics for the datasets used for evaluation (training
sets), against Wikipedia text of comparable token count sizes. Tokenization
performed with GPT-2.

###  3.1 Analysis

We begin with an analysis of the datasets and their subword properties in
order to gauge the direct effects of tokenization on the dataset’s genre and
domain. In Table 3, we present surface-level statistics reflecting the
challenges posed by the datasets’ sources, comparing each task’s training set
with comparably-sized sets of sentences sampled uniformly from English
Wikipedia. We use the GPT-2 tokenizer, which boasts the largest subword
vocabulary of all models considered in this work, to obtain a lower bound on
the added token mass expected by our models on these datasets, and report the
following measures: the number of unique word types and type/word ratio, the
percent of types which are subword-tokenized by GPT-2 (Multitoks), and the
overall increase in number of tokens over the corpus compared to a single-
token-per-word representation (i.e., strictest application of XRayEmb
considered in this work).

We find a striking disparity between the well-edited Wikipedia corpora,
themselves far from being completely in-vocabulary, and the datasets at hand.
Wikipedia itself proves to be scale-invariant on the metrics that are not TTR,
maintaining a token-OOV rate of roughly 9% and a tokenization overhead of
$\sim$13%.777This number is very close to the one found by Ács et al. (2021)
using multilingual transformer models; other languages’ corpora boast
overheads ranging from 28% (French) to 95% (Japanese). A control dataset in
the form of CoNLL-2003’s English NER portion Tjong Kim Sang and De Meulder
(2003), extracted from newswire text, exhibits a marked increase in complex
words, possibly mostly named entities, or time-sensitive terms compared to the
tokenizer’s training set, alongside a decrease in type count which reflects
its narrow source domain. NYTWIT, despite also being extracted from newswire
text, is sourced from news items that contain novel forms and so are often
written in a high register or involve niche domains; as a result, it contains
a larger overall token mass, split rather evenly across individual words
(i.e., unknown words have fairly regular structure). The Twitter NER datasets,
exhibiting tweet language, do not exceed CoNLL’s multitoken type proportion by
much, but its OOVs tend to be completely unexpected forms, leading to a much
higher raw post-tokenization count. In the emoji dataset, which has not been
pre-processed according to NER standards and instead was directly scraped off
Twitter, almost a third of all unique forms are multi-token, and their
presence enlarges the total token count by nearly two thirds. MarcoQA data,
most of which is text from highly-ranked web pages, but also including user-
generated queries, assumes a middle position between these two extremes.

##  4 Experiments

We evaluate the effect of XRayEmb when included in various LPLMs. We choose
BERT-base-cased Devlin et al. (2019), GPT2-small Radford et al. (2019), and
RoBERTa-base Liu et al. (2019) as the base LPLMs to be manipulated.888Models
loaded from the Huggingface repository Wolf et al. (2020). The former contains
roughly 108M parameters, and the latter two roughly 125M, a difference
accounted for by their larger subword vocabulary (50k vs. 29k) and the
resulting larger embedding table. All models are case sensitive, but they
differ in their strategy for preserving the original space-delimited word
sequence: BERT’s tokenizer marks word-non-initial tokens with a ## string,
while GPT-2’s tokenizer marks spaces with a special underline character and
appends them to the following word. The difference manifests itself in
sequence-initial words, whose initial token in the GPT-2 representation shares
the form of sequence-medial word-medial tokens, rather than that of sequence-
medial word-initial tokens; and in symbols pre-tokenized without a preceding
space, such as punctuation and apostrophes. RoBERTa’s tokenizer adopts the
GPT-2 marking strategy but avoids the first pitfall by internally prepending
all input sequences with a space character. We perform the 2pt phase for each
model on a collection of English tweets from 2016 obtained from the Firehose
and preprocessed to replace all ‘@’-mentions with @user. We later ablate this
domain change effect by training models with 2pt text from the English
Wikipedia March 2019 dump (see §4.3). We sample both resources to create pre-
training corpora of roughly 725MB (unzipped), several orders of magnitude
smaller than what contemporary models use for the first pre-training phase,
and train for a single epoch.

In preliminary experiments on several character-level XR-Enc architectures, we
found that a convolutional net outperforms bidirectional LSTMs and small
transformer stacks. We pass the input characters through three separate
convolution layers of width 2, 3, and 4 (characters), then pass the outputs
through max-pooling layers and a ReLU activation, and finally project the
concatenation of the results onto the base models’ embedding dimension. This
XR-Enc component contains $\sim$1M parameters, negligible compared to the
transformers’ parameter count.999The Wikipedia-trained models contain
$\sim$2.8M parameters, the difference owing to language-ID filtering performed
on the Twitter data, leading to a much smaller character set. We implement XR-
Dec as a 2-layer unidirectional LSTM whose hidden layer is initialized by
projecting the context vector $\bm{h}_{i}$ output from $M$ into the hidden
dimension. Characters are generated by projecting the LSTM’s output through
two linear layers with a tanh activation.

During 2pt, we insert a cycle dependency batch every 5,000 LM steps. For the
replacement policies we choose $\pi^{t}_{(l)}$ to sample uniformly random sets
of tokens representing 15% of space-delimited words to pass as a target loss
for XR-Enc, and $\pi^{t}_{(u)}$ to replace the embedding input to $M$ with XR-
Enc’s output for all multi-token words. $\pi^{g}$ selects all tokens to pass
as target losses for XR-Dec, calculated as a sum of the cross-entropy loss for
each character in the target sequence. Cycle batches consist of sampling $k$
words out of the $K$ most frequent words from the training corpus, with
replacement, and $k$ vectors from a Gaussian distribution blown to concentrate
around the surface of the unit sphere in hidden-dimension space:

$\tilde{\bm{e}}\sim\frac{1}{\sqrt{d}}\cdot\mathcal{N}(0,\bm{I}^{(d)}).$

E$\rightarrow$D loops are optimized for a character-level cross-entropy loss,
whereas D$\rightarrow$E loops target a euclidean distance loss. When a XR-Enc
embedding correspond to multiple $\tau$ tokens, the learning target is created
by max-pooling their embeddings.101010This $\operatorname*{agg}$ function
outperformed average-pooling and first-token selection in preliminary
experiments. We set $K=$25,000 and $k=$1,000.

###  4.1 Intrinsic Assessment

Across all base models and both 2pt corpora selected for our experiments, we
observed a steady decrease in the LPLM models’ built-in loss metrics (masked
prediction / autoregressive prediction) until stabilizing at roughly half the
initial value before the end of the 2pt epoch. This indicated that the
transformer layers are able to process inputs from both the embedding table
and XR-Enc and reconcile them. Figure 2 depicts the parameter updates in
RoBERTa by parameter type, across layers, comparing parameter values before
and after the 2pt phase on Twitter data. It shows that the change along the
model layers is fairly stable, with mildly more extensive updates in the
bottom and top layers. The former is to be expected given the introduction of
inputs from XR-Enc; the latter can also be influenced by encountering Twitter
data, which is substantially different than what RoBERTa is used to.

Figure 2: Euclidean distance between RoBERTa weight parameter values before and after a 2pt training phase on the Twitter corpus. Model |  | BERT | GPT-2 | RoBERTa
---|---|---|---|---
Corpus |  | Wikipedia | Wikipedia | Twitter | Twitter | Twitter
Steps |  | 9,000 | 13,500 | 21,000 | 7,500 | 57,500
Sequences ($10^{3}$) |  | 5,184 | 7,776 | 10,080 | 2,160 | 16,560
|  | proming | crordman | d | orereren | everyone
|  | dy | sssion | . | ant | kerned
|  | deded | gental | the | re | levernger
|  | terse | 2 | @ | cerent | and
|  | h | ther | ==666!!!!!!!!!!!!!!! | ennte | ed
Table 4: Example generated words from random locations near the surface of the
unit sphere in $\mathds{R}^{768}$.

Another artifact of 2pt is the outputs of the XR-Dec module. While monitoring
the training procedure, we periodically sample words from random locations
centered around the surface of the unit sphere of the embedding space, to see
what priors the generative net is learning from the vectors encountered during
training. Table 4 presents some of these samples for different models and
different corpora at different points in the training phase. Masked models
appear to be learning well-formed fractions of English or pseudo-English words
at early stages of the training phase from both Wikipedia and Twitter data. As
training continues, fewer sequences containing repetitions are observed, fewer
generations occur across samples, and more in-vocabulary words appear,
suggesting convergence of the vector space towards representing well-formed
and diverse English vocabulary. The autoregressive GPT-2, on the other hand,
struggles to produce meaningful sequences beyond short words and punctuation
symbols when trained on the informal Twitter input, suggesting a difficulty in
learning a mapping of language from vector space without availability of a
two-sided context.

###  4.2 Downstream Evaluation

During task fine-tuning and inference, since we do not evaluate on generative
tasks, we do not use XR-Dec. We perform minimal hyperparameter search for each
base model + task combination, and fine-tune model parameters during
downstream training in all tasks but NYTWIT. For these tasks, we also
experimented with setups where $M$ and XR-Enc are used as feature extractors
only, and where XR-Enc training is supplemented by an additional embedding
loss (as described in §2.1 for the 2pt phase) computed against embeddings of
the task input. Neither setup provided improvement on any task during our
tuning experiments (see §4.3).

Downstream models are implemented as follows: for sequence classification and
ranking, a two-layer perceptron with ReLU activation is trained to make the
prediction from the top-layer representation of the initial [CLS] token (in
BERT and RoBERTa models) or of the final token in the sequence (in GPT-2). For
NER, an LSTM is run over the sequence of each word’s top-layer representation,
followed by a single linear layer which makes the prediction. For NYTWIT,
$M$’s contextualized vector for the target word is used as input for a single
logistic layer. In cases of multi-token words, the prediction from the first
token is selected. We set XR-Enc’s character embedding dimension to 200 and
the convolutional layers to 256 channels. Following Sun et al. (2019), we set
the maximum learning rate to $10^{-3}$ for the task models and $2\times
10^{-5}$ for fine-tuning, and perform warm-up for 10% of the total expected
training steps before linearly decaying the rates to zero. All parameters are
optimized using Adam Kingma and Ba (2015) with default settings. We run all
NER models for twenty epochs and sequence-level task models for three,
evaluating on the validation set after each epoch using the metrics reported
below, and stopping early if performance has not improved for four epochs. In
order to avoid unfairly favorable conditions for XRayEmb models, task
hyperparameters are all tuned on the base models, with XRayEmb models using
only values on which their base equivalents have also been evaluated.

We evaluate the effectiveness of XRayEmb’s concepts and components by
comparing the following setups:111111A setup where all words are represented
by XR-Enc obtained noncompetitive results on all tasks.

  * •

None uses only the base model;

  * •

None+2pt uses a version of the base model that was further pre-trained on the
same Twitter corpus on which XRayEmb is trained (adaptive fine-tuning Ruder
(2021)), in order to control for the increase in total unlabeled text seen by
the model;

  * •

Scaffolding is a model trained with XRayEmb in a 2pt phase, but only using
base model embeddings during task fine-tuning;

  * •

Stochastic samples 10% of the words in the downstream datasets, calling XR-Enc
on their character sequences while using the base model’s embedding(s) for the
remaining 90%;

  * •

All no-suff calls XR-Enc on all multi-token words which are not of the form
[token suff], where suff is a member of the Suffixes set described in §2, and
uses the base model’s embedding on the rest.

Base | XRayEmb | Emoji | Twitter NER | Emerging NER | QA | NYT- | Avg.
---|---|---|---|---|---|---|---
|  | Dev | Test | Dev | Test | Dev | Test |  | WIT | Test
BERT | None | 24.30 | 37.29 | 29.06 | 28.66 | 38.14 | 29.01 | 46.78 | 45.21 | 37.39
|  None+2pt | 29.25 | 39.40 | 32.05 | 29.50 | 39.79 | 29.25 | 50.02 | 44.87 | 38.61
| Scaffolding | 28.17 | 38.60 | 32.45 | 31.38 | 41.10 | 29.25 | 49.50 | 53.62 | 40.47
| Stochastic | 27.54 | 37.41 | 30.77 | 27.51 | 37.99 | 27.98 | 48.12 | 49.80 | 38.16
| All no-suff | 27.12 | 38.63 | 34.09 | 29.59 | 39.47 | 28.43 | 48.64 | 34.76 | 36.01
GPT2 | None | 24.89 | 38.72 | 30.25 | 26.70 | 40.06 | 28.40 | 44.90 | 47.25 | 37.19
|  None+2pt | 25.47 | 40.95 | 32.00 | 28.51 | 40.68 | 29.90 | 47.18 | 47.44 | 38.80
| Scaffolding | 25.29 | 40.70 | 30.46 | 28.97 | 41.16 | 28.23 | 46.99 | 50.99 | 39.18
| Stochastic | 25.23 | 38.55 | 32.20 | 28.10 | 39.58 | 28.35 | 46.11 | 46.37 | 37.50
| All no-suff | 26.30 | 34.27 | 36.27 | 32.48 | 49.20 | 34.87 | 44.98 | 33.21 | 35.96
RoBERTa | None | 25.07 | 39.50 | 48.57 | 44.86 | 56.43 | 46.22 | 45.12 | 48.31 | 44.80
|  None+2pt | 27.04 | 42.82 | 47.39 | 43.84 | 57.13 | 45.12 | 48.98 | 48.43 | 45.84
| Scaffolding | 25.87 | 41.12 | 49.71 | 45.51 | 56.28 | 44.60 | 48.26 | 53.98 | 46.69
| Stochastic | 26.38 | 40.09 | 49.57 | 44.82 | 58.21 | 45.26 | 48.54 | 51.80 | 46.10
| All no-suff | 26.82 | 33.07 | 46.55 | 42.72 | 55.07 | 43.91 | 47.45 | 35.65 | 40.56
SOTA (reported) |  | 47.46 |  |  52.40 |  |  49.60 |  |  48.40 |
Table 5: Results on all models (Emoji, NER, NYTWIT: Micro-F1 $\times$ 100; QA:
MRR $\times$ 100), all results except for None are averaged over three models
initialized on different random seeds. Best result for each base model in
bold, best across all underlined.

We present the results of the downstream prediction tasks in Table 5. All
transformer models except for None were 2nd-phase pre-trained three separate
times using different random seeds, and the mean results are reported.

The first observation we make is the dominance of RoBERTa, a thoroughly
optimized masked language model, over the other models on the NER datasets in
all its variants. This suggests RoBERTa has captured fine-grained information
about individual words that it was able to retain in its representations for
them; the struggle of the XRayEmb model to provide improvement over the None
versions of the model strengthens this hypothesis. GPT-2, despite having
access to only left-side context of each word, still outperforms the basic
BERT model on most setups in the NER tasks, perhaps due to its larger subword
vocabulary size. At the same time, its gains from the XR-Enc representations
are much more considerable, suggesting that its left-context-only inference
may in fact be detrimental to the its $M$’s performance as a whole. In
general, models perform better on Emerging NER than on Twitter NER, which we
attribute to several possible causes or their combination: first, the source
shift in the Emerging NER test set from Twitter to Reddit is meant to encumber
the models, but given the extensive pre-training they undergo they may
actually benefit from the fact that test sequences are longer on average than
those in the Twitter NER dataset; second, more prosaically, the Emerging NER
dataset contains fewer entity types (6 vs. 10), making the task itself
somewhat easier.

The other word-level task, NYTWIT, demonstrates substantial gains made by the
XRayEmb training regime: the Scaffolding setup preforms best in all three base
models, and in both masked langugage models the Stochastic setup outperforms
both None variants. Together with the NER results, this suggests that XRayEmb
succeeds in providing transformer models with word-level representations that
allow coarse-grained classifications (such as named entity type or novel word
origin), better than default subword segmentations, for both edited and user-
generated text.

We find that XRayEmb is less successful in improving sequence classification
and ranking performance than on word-level tasks. The None+2pt setup obtains
the best results in most models on the Emoji and QA datasets, suggesting the
improvements seen in XRayEmb-based models is mostly attributable to the domain
shift introduced by the Twitter pre-training corpus.121212We note the complete
inconsistency in both model performance and comparative model ranking present
in the Emoji dataset, which calls back the systemic issue we identified in §3:
The time span over which the test set was collected contained a fundamental
shift in Twitter’s properties — doubling the tweet length limit from 140 to
280 characters — and so exhibits an unpredictable corpus incompatible with the
train and dev partitions. As a result, model performance over the dev set does
not predict the test set results (a large difference on performance in this
task between dev and test sets was also observed in macro-F1 scores by
Barbieri et al. (2020)). In fact, under such specific data shift
circumstances, it could be the case that the simpler base model is better
equipped for facing longer test data, which scales generalizations made over
the training and dev sets, as opposed to XR-Enc-augmented models which have
more levels of generalization to acquire during training and cannot anticipate
the scale change. Post-hoc inspection of specific model outputs resulted in
some more concrete hypotheses for the cause of discrepancy in the major
categories of confusion, for example the distributions of @ presence in heart
emoji tweets and heart-eyes emoji tweets shifted to a degree which could
explain the models’ growing confusion between the two, but no signals
accounted for the entire difference in performance and we conclude that the
main reason remains the change in sequence length distributions. This suggests
that XR-Enc may be a good learner for word-internal phenomena, picking up
structural cues as to their roles within or without context, making it more
useful locally than subword tokens’ uninformed embeddings, while not being
strong enough to provide a better semantic prior for $M$ to aggregate together
with surrounding well-formed words, making its global utility limited.

Ablation | BERT | GPT2 | RoBERTa
---|---|---|---
Full | 37.17 | 38.27 | 41.97
No FT |  $-$0.31 |  $-$0.25 |  $-$0.80
Wiki 2pt |  $-$3.11 |  $-$2.67 |  $-$2.17
No loops |  $+$0.82 |  $+$0.26 |  $-$1.12
All multi |  $-$1.39 |  $+$0.44 |  $-$1.81
Table 6: Dev set average effect of model variants compared with the All no-
suff condition: No-FT — pre-trained model used only for feature extraction;
Wiki — XRayEmb trained on Wikipedia data instead of Twitter; No loops —
trained without the cycle dependency loops; All multi — common-suffix words
also inferred using XR-Enc.

###  4.3 Ablations

We compare the dev set results of the All no-suff condition on several
modified versions of the model, presented in Table 6. First, we find that
fine-tuning $M$ and XRayEmb parameters during downstream task application is
beneficial for results across base models, indicating susceptibility of XR-
Enc’s network to tune itself on task data and not solely on LM and the
vectorization signal. Next, and most substantially, we note the vast
improvement of Twitter-trained models compared with 2pt performed over a
Wikipedia corpus with comparable size; even though some tasks are on edited
text, the overall effect of domain change during second pre-training is
apparent (and, indeed, least impactful on the NYTWIT task which features the
best-edited text). Other decisions made during the 2pt phase appear to be less
decisive: removing the dependency loops helps performance on BERT and GPT2,
but makes a large dent in the best-performing RoBERTa, indicating potential
gains to be made by applying XR-Dec in generative tasks not pursued within our
scope; the suffix-based dialing down of inference in pre-training helps the
masked models but hurts GPT2 performance, possibly because its autoregressive
application prohibits it from looking at a simply-inflected word’s suffix when
processing its stem, a problem not incurred in masked modeling.

##  5 Related Work

CharBERT Ma et al. (2020b) is a method which incorporates character-level
encodings in LPLMs, while re-designing the transformer stack to include a
character channel distinct from the parallel token channel. Char2Subword
Aguilar et al. (2020) integrates multiple losses in a system trained to
predict subword tokens from the character level, all originating in distance
metrics between target and prediction. CharacterBERT El Boukkouri et al.
(2020) uses an ELMo-style character convolutional network to encode input into
a transformer MLM. CANINE Clark et al. (2021) is a method for training LPLMs
which removes the need for a subword tokenizer, by utilizing character-level
representations which are pooled into inputs for the main transformer module.
ByT5 Xue et al. (2021) ventures yet deeper by training a T5-architecture model
Raffel et al. (2020) over byte representations, reducing the required
embedding table size by one more order of magnitude. Unlike XRayEmb, these
systems all require training an LPLM from scratch and do not support using a
2pt step for tuning existing large models. In addition, with the exception of
CANINE, they all resort to softmax prediction over a subword vocabulary for
the generative portion of the pre-training phase, and do not offer a
character-level decoder which can also be tied to the encoding component.

##  6 Conclusion

We present XRayEmb, an adaptive encoding-decoding system which allows large
pre-trained language models to handle representation of words learned
compositionally from their orthographic manifestation without resorting to
hard-coded subword embeddings which fare poorly on the thick distribution
tails of domain-shifted data, while still benefiting from token
representations learned during the models’ original pre-training phase. We
demonstrate XRayEmb’s efficacy across three pre-trained model architectures
when trained on a corpus from a domain new to them, showing that with a
relatively small amount of processing improvements can be reached on word-
level tasks; with sequence-level task performance holding out in powerful pre-
trained systems. In future work, we wish to extend the scope of XRayEmb to
non-English languages and to multilingual models, as well as to make use of
the XR-Dec module to assist in generative tasks.

## Acknowledgments

We thank Jon Clark, Dan Garrette, Mark Riedl, Diyi Yang, Dan Roth, and Wei Xu
for their helpful comments on earlier drafts.

## References

  * Ács et al. (2021) Judit Ács, Ákos Kádár, and András Kornai. 2021.  Subword pooling makes a difference.  _arXiv preprint arXiv:2102.10864_.
  * Aguilar et al. (2020) Gustavo Aguilar, Bryan McCann, Tong Niu, Nazneen Rajani, Nitish Keskar, and Thamar Solorio. 2020.  Char2subword: Extending the subword embedding space from pre-trained models using robust character compositionality.  _arXiv preprint arXiv:2010.12730_.
  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  TweetEval: Unified benchmark and comparative evaluation for tweet classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1644–1650, Online. Association for Computational Linguistics.
  * Barbieri et al. (2018) Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018.  SemEval 2018 task 2: Multilingual emoji prediction.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 24–33, New Orleans, Louisiana. Association for Computational Linguistics.
  * Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007.  Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 440–447, Prague, Czech Republic. Association for Computational Linguistics.
  * Chen et al. (2018) Jing Chen, Dechuan Yang, Xilian Li, Wei Chen, and Tengjiao Wang. 2018.  Peperomia at SemEval-2018 task 2: Vector similarity based approach for emoji prediction.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 428–432, New Orleans, Louisiana. Association for Computational Linguistics.
  * Clark et al. (2021) Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021.  Canine: Pre-training an efficient tokenization-free encoder for language representation.  _arXiv preprint arXiv:2103.06874_.
  * Coster et al. (2018) Joël Coster, Reinder Gerard van Dalen, and Nathalie Adriënne Jacqueline Stierman. 2018.  Hatching chick at SemEval-2018 task 2: Multilingual emoji prediction.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 445–448, New Orleans, Louisiana. Association for Computational Linguistics.
  * Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017.  Results of the WNUT2017 shared task on novel and emerging entity recognition.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * El Boukkouri et al. (2020) Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020.  CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6903–6915, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Klein and Tsarfaty (2020) Stav Klein and Reut Tsarfaty. 2020.  Getting the ##life out of living: How adequate are word-pieces for modelling complex morphology? In _Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 204–209, Online. Association for Computational Linguistics.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Sebastian Ruder, Dani Yogatama, et al. 2021.  Pitfalls of static language modelling.  _arXiv preprint arXiv:2102.01951_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2020a) Weicheng Ma, Ruibo Liu, Lili Wang, and Soroush Vosoughi. 2020a.  Multi-resolution annotations for emoji prediction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6684–6694, Online. Association for Computational Linguistics.
  * Ma et al. (2020b) Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020b.  CharBERT: Character-aware pre-trained language model.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 39–50, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  In _CoCo@ NIPS_.
  * Peng et al. (2018) Hao Peng, Sam Thomson, and Noah A. Smith. 2018.  Backpropagating through structured argmax using a SPIGOT.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1863–1873, Melbourne, Australia. Association for Computational Linguistics.
  * Pinter et al. (2020) Yuval Pinter, Cassandra L. Jacobs, and Max Bittker. 2020.  NYTWIT: A dataset of novel words in the New York Times.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6509–6515, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _arXiv preprint_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Ruder (2021) Sebastian Ruder. 2021.  Recent Advances in Language Model Fine-tuning.  http://ruder.io/recent-advances-lm-fine-tuning.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Strauss et al. (2016) Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine de Marneffe, and Wei Xu. 2016.  Results of the WNUT16 named entity recognition shared task.  In _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_ , pages 138–144, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune BERT for text classification?  In _China National Conference on Chinese Computational Linguistics_ , pages 194–206. Springer.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 30:5998–6008.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021.  Byt5: Towards a token-free future with pre-trained byte-to-byte models.  _arXiv preprint arXiv:2105.13626_.