marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

RetVec: Resilient and Efficient Text Vectorizer

Elie Bursztein 1  Marina Zhang 1  Owen Vallis 1  Xinyu Jia 1  Alexey Kurakin 1

††footnotetext:  1Google, Mountain View, USA. Correspondence to: Elie
Bursztein<EMAIL_ADDRESS>
Under submission

###### Abstract

This paper describes RetVec, a resilient multilingual embedding scheme
designed for neural-based text processing, including small-text classification
and large-language models. RetVec combines a novel character encoding with an
optional small model to embed words into a 256-dimensional vector space. These
embeddings enable training competitive multilingual text models resilient to
typos and adversarial attacks. In this paper, we evaluate and compare RetVec
to state-of-the-art tokenizers and word embeddings on common model
architectures. These comparisons demonstrate that RetVec leads to competitive
models that are significantly more resilient to text perturbations across a
variety of common tasks. RetVec is available under Apache 2 license at
https://github.com/[anonymized].

##  1 Introduction

Researchers have proposed several techniques to convert text into dense
representations suitable for training neural networks, from the early use of
character bigrams and trigrams to the more recent creation of word semantic
embeddings and the development of sub-word tokenizers. Each of these
approaches attempt to mitigate issues caused by OOV (Out Of Vocabulary)
tokens, including intentional (adversarial attacks) and unintentional (e.g.,
typos or unseen words during training) text perturbations. These approaches
leverage techniques such as sub-word tokenization (e.g., SentencePiece Kudo &
Richardson (2018)) and learning smooth representations (e.g., FastText
Bojanowski et al. (2017)).

However, we find that each of these approaches suffers from at least one of
following drawbacks:

  * •

Insufficient resilience against typos and adversarial attacks.

  * •

Requiring large dictionaries and embedding lookup tables.

  * •

Poor performance on certain languages or in multilingual settings.

RetVec addresses these issues by combining a novel UTF-8 character encoder
with an optional small model (200k parameters) that projects encoded words,
including typo-laden ones, into a compact 256 dimensional syntactic metric
embedding as detailed in section 3.

We train our syntactic metric embedding using pair-wise learning which ensures
that typo-laden words are close in the embedding space to their original
words. Models trained with our embedding are competitive and significantly
improve the performance on typos and adversarial attacks over other popular
tokenizers and word embeddings.

Furthermore, as RetVec works directly on UTF-8 characters, it does not require
any dataset pre-processing or additional embedding layers. This property
allows RetVec-based multilingual models to train in a rapid manner. RetVec is
also space efficient as it does not require shipping with a large embedding
lookup table, only a small optional model (1MB). This choice reduces model
memory footprints, making RetVec-based models ideal candidates fore deploying
on-device and on other edge devices where storage, bandwidth, and memory
resources are scarce.

Through a series of extensive experiments, we demonstrate that RetVec
outperforms or is comparable to other state-of-the-art tokenizers and word
embeddings including BPE Sennrich et al. (2016), SentencePiece Kudo &
Richardson (2018), and FastText Bojanowski et al. (2017).

Our paper makes the following contributions:

  * •

We introduce RetVec, a resilient embedding scheme designed for neural-based
text processing based on a novel character encoder.

  * •

Due to it’s novel encoder RetVec works seamlessly on every combination of
languages without the need to adapt or retrain it.

  * •

We show that RetVec is faster than other embedders on multi-core CPU and on
GPU. Additionally, we show that RetVec uses significantly less memory than the
other embedders.

  * •

We compare RetVec’s classification accuracy, resilience to typos, and
adversarial attack resilience against other tokenizers and embeddings on
common architectures for a wide range of datasets. We show that models trained
with RetVec have slightly higher accuracy, greater resilience to typos, and
significantly better resilience to adversarial attacks than models trained
with the other tokenizers and embeddings.

  * •

We provide a TensorFlow implementation of RetVec, including its pret-rained
models, alongside the code to reproduce our benchmarks under the Apache 2
license at https://github.com/[anonymized].

##  2 Background

Deep neural networks (DNNs) operate on floating-point tensors. To process
textual information, DNNs have to convert text into a floating-point
representation. This is typically achieved in the following way:

  * •

First, a tokenizer splits text into tokens. These could be words, pieces of
words or even individual characters.

  * •

Then, each token is encoded as a floating point vector using an embedding
algorithm.

#### Tokenizers

Text tokenizers play a major role in text-model accuracy by ensuring that text
is properly segmented into tokens which are typically words or sub-words.

Simple tokenizers split entire texts into words using whitespace and
punctuation as splitting points. More sophisticated tokenizers (such as BPE
Sennrich et al. (2016) and SentencePiece Kudo & Richardson (2018)) are
typically used in modern LLMs (Large Language Models) to split text into sub-
words to reduce vocabulary size and out-of-vocabulary (OOV) issues by
representing unknown words as combinations of known tokens. SentencePiece
deals better with segmentation and OOV, and is used by most state-of-the-art
models these days Raffel et al. (2020); Chowdhery et al. (2022). Both BPE and
SentencePiece require a training phase to adapt to the targeted dataset and
require shipping extra files with the models for inference purposes.

#### Word embeddings

After text is tokenized, each token is encoded as a floating point vector of
fixed dimensionality. This process, called embedding, is often done by
dictionary lookup into an embedding table. The idea behind building an
embedding table is to learn to project words with related meaning next to each
other within a metric space. This can either be achieved by using an
independently pre-trained embedding lookup table using algorithms such as
Word2Vec Mikolov et al. (2013), Glove Pennington et al. (2014) and fastText
Bojanowski et al. (2017), or by randomly initializing an embedding table and
training it as a part of the LLM (like GPT-2 Bet (2019)). Note that embedding
table lookup is typically limited to known, in-vocabulary tokens and will
replace all OOV tokens with the same OOV token or hash the token into a fixed
array of tokens. This significantly reduces the representational power of such
embedding models in the presence of typos and character-level perturbations.
Additionally, embedding lookup tables can be very large. For example, the
embedding table for the multilingual BERT-Base model occupies around 350
MiB111 120k tokens in vocabulary * 768 embedding dimension * 4 bytes per
float.

In this paper, we build on this idea of projecting words into a metric space
to pre-train RetVec optional models; however, instead of projecting words next
to semantically related words, we project them next to typo-laden versions of
themselves to increase the tokenizer resilience to text augmentation and to
reduce OOV issues. In section 3.3 we discuss use of a heavily augmented
version of the FastText multilingual corpus Wang et al. (2020) to pre-train
RetVec models, and in section 4 we compare RetVec’s efficiency to fastText and
other tokenizers.

#### Pair-based learning

Pair-based learning aims to learn an embedding space, where embedded vectors
of similar samples are encouraged to be closer while dissimilar ones are
pushed apart from each other Xing et al. . This approach has been successfully
applied to various tasks including image retrieval He et al. (2018), face
recognition Schroff et al. (2015), and zero-shot learning Zhang & Saligrama
(2016). In the deep learning setting, this type of learning is accomplished
using loss functions that minimize/maximize the pairwise distances within the
embedding space. In this paper we experiment with Multi Similarity loss Wang
et al. (2020) and Circle loss Sun et al. (2020a), two of the most recent and
most efficient pairwise losses.

As discussed in detail in section 3.3, we use a pairwise loss to pre-train
RetVec optional models by pulling words close to their typo-laden version and
away from other words. This training objective is empirically the best-suited
for increasing the resilience of RetVec embeddings against typographical
augmentations and to reduce OOV issues such as those evaluated in section 9.

#### Typo resilience

One challenge faced by text tokenizers is the existence of typographical
errors that might not appear in the training dataset. At inference time, these
typographical errors can lead the tokenizer to incorrectly output OOV ids for
valid words. This behavior ultimately leads to lower accuracy, particularly
for retrieval tasks where human queries are known to contain typos Zhuang &
Zuccon (2021). Query datasets compiled from real-world logs exhibits a
typographical error rate between 16% and 19% Hagen et al. (2017). Furthermore,
text models are vulnerable to adversarial attacks that intentionally corrupt
words to force the model to predict wrongly. Many character-level adversarial
attacks aim to maximize the loss of accuracy per word manipulation by
triggering the most detrimental OOV cases. In recent years, many successful
adversarial text attacks Morris et al. (2020) have been developed, showcasing
the vulnerability of text models to these types of augmentations. These
attacks can be viewed as the most severe form of intentional typos. To
demonstrate RetVec ’s goal resilience to adversarial typographical
augmentation, we report the model’s performance under the three popular
character based text adversarial attacks: TextBugger Li et al. (2019), WordBug
Gao et al. (2018)and Pruthi Pruthi et al. (2019) in section 7.

##  3 RetVec

In this section we describe how RetVec works. We start by detailing its novel
character encoder. Then, we discuss its optional model’s design and key
architectural decisions. Finally, we discuss the training procedure used to
train and optimize those optional models.

Figure 1: RetVec general architecture overview - output shape of each layer in
parenthesis. ‘clen’ indicates the number of characters used per word. Batch
and sequence length dimensions are omitted.

###  3.1 Character encoder

RetVec character encoder, as shown in figure 1 is composed of the two lowest
layers of RetVec and is the core innovation that allows RetVec to statelessly
encode every language. The overall idea being to convert characters to their
UTF-8 int32 representation (Integerizer) and then convert those into their
binary representation (Binarizer). The number of characters considered by word
is fixed to create a dense vector. As discussed in the ablation section 9, we
found that 16 characters per word empirically works very well for most cases,
and using 32 characters (99.5% of all words in the FastText corpus) provides
no accuracy gains.

The Integerizer is similar to the one used in ByT5 Xue et al. (2022) which
also encodes characters to their UTF-8 codepoint. What makes RetVec unique is
that it then encodes those integers into their binary representation instead
of one-hot encoding them via its Binarizer. Converting to binary
representation creates a significantly more compact representation while
preserving its learnability. We tested a lot of other approaches to convert
the Integerizer output into more compact forms, including trying to use a
single float per character by dividing the UTF-8 integer value by the max
codepoint in the Basic Multilingual Plane, but these more compact
representations didn’t appear to be learnable.

Importantly, the Integerizer and the Binarizer are kept as separate
functions/layers for efficiency reasons: when using a remote accelerator such
as TPU, it is advantageous latency-wise to send the output of the Integerizer
(e.g $16\times$ int32) across the network rather than the Binarizer output
which is significantly larger (e.g $16\times 32$ float32). The binarization in
that setup is performed by the initial part of the model as a layer.

We note that while TensorFlow doesn’t have native support for converting
integers to binary, we found out that, as reported in the performance
evaluation in Section 4, writing it using bit shifts with a precomputed matrix
to be sufficiently fast to not be a bottleneck. If RetVec becomes widely used,
adding binarization native support in deep-learning frameworks will improve
its speed further.

###  3.2 Optional model architecture

While RetVec provides a stateless, efficient word representation using its
Integerizer and Binarizer, directly using those word representations is not as
performant as state-of-art tokenizers. The RetVec embedding size is also 512
dimensions, which is larger than it needs to be. To address these
shortcomings, and to make the RetVec embeddings more efficient and resilient,
we add a small optional model on top of the Binarizer output that compresses
the output to a smaller embedding (e.g. 256 dimensions per word).

Overall, we found that models trained with versions of RetVec that include
this optional model have an average higher accuracy than models trained using
other tokenizers (see sec 5), and reduces the adversarial attack success rate
by an average of 7% compared to the base version of RetVec without an optional
model; however, adding this optional model also incurs a small drop in
tokenization speed. This makes adding the optional model a good tradeoff
between performance gains and computational cost.

| Architecture |  | Embedding
---
Size
| Num
---
Parameters
| Pre-train
---
Loss
| Average
---
Classification
Accuracy
| Average Mixed Typo
---
Resilience Accuracy
(30% corruption)
| Average Adversarial
---
Attack Success Rate
RetVec-raw | - | 512 | 2,048 | N/A | 88.7% | 84.7% | 44.5%
RetVec-base | Dense | 256 | 201,536 | 0.0029 | 89.6% | 87.3% | 37.4%
RetVec-large | Transformer - GAU | 256 | 148,451 | 0.0024 | 89.7% | 87.4% | 39.8%

Table 1: The three variations of RetVec considered in this paper. The RetVec-
raw parameters come from the BatchNormalization layer added to models trained
with this embedding as it greatly improves their convergence and performance.

As summarized in table 1, we propose three versions of RetVec that exhibit
different performance and computation tradeoffs:

RetVec-raw: This version only uses the RetVec novel character encoder and a
batch normalization layer to speed up convergence. It is the fastest model but
also the least resilient and performant of the three.

RetVec-base: This version, which is our baseline configuration, uses a small
optional model consisting of a few dense layers. It offers state of the art
performance for a minimal increase in computational cost.

RetVec-large: This version, which uses the FLASH transformer architecture Hua
et al. (2022), pushes performance as much as possible at the expense of
computational cost. As reported in table 1 the gains are marginal and
therefore best suited for applications that are very performance sensitive.

We report more detailed classification benchmarks in section 5, resilience in
section 6, and adversarial resilience in section 7. Performance overhead
details are available in section 4.

The following paragraphs provide a brief summary of the key architectural
decisions made for the RetVec models. Additionally, we provide an ablation
study that highlights the impact of the different parameters in section 9 and
also provide the full list of model parameters in appendix (table 16).

RetVec-base uses three dense layers whereas RetVec-large uses two GAU
transformer layers Hua et al. (2022). RetVec-base uses GeLU activation and
RetVec-large uses Swish activation. Both models truncate words to 16
characters maximum. To train, we use Adam with Cosine Decay as the optimizer
along with Multi-Similarity Loss Wang et al. (2020). All our models are pre-
trained on a typo augmented version of the FastText datasets Grave et al.
(2018). Our pre-training procedure is detailed in section 3.3.

We initially expected to use the transformer architecture as the base model,
as the transformer model pre-trained loss was significantly lower than the
dense model (see table 1). We even trained a larger transformer model that
took 32 characters as input and had 384 outputs. However, as reported in table
1 and detailed in the evaluation sections 5, 7, and 6, lower model pre-
training loss doesn’t always translate into increased accuracy for models
using RetVec as their tokenizer. This surprising lack of correlation made it
challenging to hypertune models as we had to evaluate them by training
classification models, adding complexity and noise to the hypertuning. We
discuss the details of our pre-training procedure that addresses this
challenge in section 3.3.

Additionally, we tried increasing the capacity of both RetVec-base and RetVec-
large by using more layers or making the layers wider (e.g increasing the
number of dimensions), but as discussed in the ablation study 9, increasing
capacity didn’t empirically translate into any meaningful gains. In hindsight,
this finding seems to be inline with the fact that other word embedding such
as Word2Vec Mikolov et al. (2013) and Glove Pennington et al. (2014) use very
shallow models consisting of only a single layer.

###  3.3 Pre-training procedure

#### Dataset

RetVec models are pre-trained using a heavily typo-augmented version of the
157-language fastText common words datasets Grave et al. (2018) extracted from
common crawl Grave et al. (2018). For each of the 88.8M deduplicated words
available in the corpus, we programmatically generate 10 rounds $\times$ 10
variations per word (2 $\times$ exact words $+$ 8 augmented words) giving us a
total of about 8.88B words to train on.

For the initial models development and ablation study we use a single dataset
round (approx. 888M words) to reduce computation cost and carbon footprint.
Using a single round ensures the model sees each unique word in our dataset
two times along with eight typo-augmented variations.

Training on this reduced dataset leads to a training loss of 0.0054 up from
0.0029 for RetVec-base and a loss of 0.0043 up from 0.0024 RetVec-large. The
classification models that use RetVec-base trained on the reduced dataset
suffered an average loss of about 0.2% accuracy on our classification
benchmarks.

#### Augmentations

The maximum human error frequency rate commonly observed is around 20%, which
is why we corrupt each word up to 25% by randomly inserting up to 4 typos
Hagen et al. (2017). We use 22 distinct types of typo augmentation that can be
grouped into the six categories of character manipulation found in search
query datasets: deletion, insertion, delimiters, missing or wrong special
characters, substitution, and transposition Hagen et al. (2017). The full list
of augmentation used is reported in appendix D. Additionally, we experimented
with supplementing the dataset with random strings and other augmentation
ideas but none of them translated to meaningful improvements.

#### Loss

We experimented with various types of losses, including: reconstruction loss
such as MSE to reconstruct the original word; contrastive loss such as SimSiam
Chen & He (2020); token replacement detection and pair-based losses like
Multi-Similarity Loss Wang et al. (2020) and Circle Loss Sun et al. (2020b).
Overall, as reported in appendix F, we found that pair-based losses work best
and combining them with other losses does not empirically bring any
improvement for neither RetVec-base nor RetVec-large.

#### Evaluation

As alluded to earlier, an unforeseen key challenge we discovered while
developing RetVec models is that a lower loss didn’t directly translate into
meaningful performance for the models using RetVec as tokenizer. To overcome
this challenge we had to include an extra validation step that trains and
evaluates models on the Multilingual Amazon Reviews Corpus McAuley & Leskovec
(2013) polarity classification task to measure downstream improvement. This
validation step is described in more detail in the evaluation section 5. We
choose the Amazon dataset due to its multilingual nature and difficulty.

##  4 Experiment: Speed

|  |  |  |  |  | CPU | GPU
---|---|---|---|---|---|---|---
Name |  | Embedding
---
Size
| Vocabulary
---
Size
| Preprocess
---
Time
| Sys
---
Mem (GB)
| Model
---
Params
Wall(s) | Usage |  | Core
---
Sec
Wall(s) | Usage | Memory(GB)
SentencePiece | 256 | 32k | 743 | 10.9 | 8M | 877 | 124% | 1087 | - | - | -
BPE | 256 | 32k | 906 | 3.6 | 8M | 1062 | 169% | 1795 | - | - | -
FastText | 300 | - | 420 | 43.9 | 0 | 845 | 52% | 439 | - | - | -
Whitespace | 256 | 32k | 132 | 1.7 | 8M | 496 | 126% | 625 | 396 | 100% | 0.5
RetVec-raw | 512 | - | 0 | 1.9 | 2k | 225 | 450% | 1013 | 233 | 23% | 0.4
RetVec-base | 256 | - | 0 | 2.0 | 200K | 574 | 843% | 4839 | 283 | 35% | 0.5
RetVec-large | 256 | - | 0 | 3.0 | 148k | 2118 | 1172% | 24823 | 480 | 56% | 0.6

Table 2: List of embedders benchmarked in this paper alongside their
processing time for the Multilingual Amazon Reviews polarity dataset both in
CPU and GPU.

In this section, we evaluate how quickly RetVec can process datasets compared
to the other commonly used tokenizers with a layer on learned embedding on
top. We considered SentencePiece Kudo & Richardson (2018), BPE Sennrich et al.
(2016), simple whitespace tokenizer and fastText Bojanowski et al. (2017). We
use the popular HuggingFace Hug  implementation of BPE and the official
SentencePiece implementation throughout this paper.

#### Setup

We use the Multilingual Amazon Reviews McAuley & Leskovec (2013) dataset for
the tokenization speed evaluations. The corpus is composed of 1.2M tokens –
200k tokens for 6 languages (English, French, German, Spanish, Chinese,
Japanese).

All the experiments reported in this section are run on a standard Google
Cloud Compute machine with 16 CPU cores and a V100 Nvidia GPU. We disable the
GPU visibility to perform the CPU measurements and use the GNU time command to
measure the total wall time, CPU usage, and system memory. We use nvidia-smi
to record the GPU usage and memory by running it at a 10 second interval
during execution. The result table 2 reports a single number for the GPU usage
as we found out that the GPU usage and memory was stable throughout the entire
evaluation runs.

For fairness purposes, as visible in table 2, we report both the time it takes
to adapt to the datasets and how long it takes to tokenize the entire dataset
end-to-end as distinct numbers both for CPU and GPU for the tokenizers that
benefits from GPU acceleration namely the Learned embedding and RetVec. We
also report normalized CPU-core times as RetVec tokenizer and other tokenizer
to a lesser extent benefits from multi-threaded acceleration. Finally, we
report memory usage when applicable, both system and GPU, as those are
important metrics when considering performing classification on memory
constrained devices such as IoT and smartphones.

#### Results

We found that RetVec-raw and RetVec-base are the fastest tokenizers when a GPU
is available and are in the top three when multi-core CPU are available, which
is common on current devices including smartphones. RetVec-base’s performance
on CPU stems from its ability to efficiently use the available cores as
visible in the Core Sec column – we expect that more optimized versions of
SentencePiece and BPE could achieve similar performance. RetVec-large is also
faster than SentencePiece and BPE when GPU are available but significantly
slower (approx. 4x) than every other tokenizer when running on multi-core CPU
making its performance/computation trade off less appealing than RetVec-base.

##  5 Experiment: Classification

In this section we report how well RetVec performs compared to other
tokenizers and word embeddings when used to train models for classification
purposes. We report classification accuracy on a wide range of datasets for
the most common type of model architecture to evaluate the genericity of the
approach.

#### Setup

Name | Train Size | Test Size | # classes | # langs | OOV words | words per examples | Avg # word length
---|---|---|---|---|---|---|---
AGNews Zhang et al. (2016) | 120k | 7.6k | 4 | 1 | 9.9% | 45 | 4.4
DBPedia Zhang et al. (2016) | 560k | 70k | 14 | 1 | 32.6% | 55 | 4.6
Yelp Zhang et al. (2016) | 560k | 38k | 2 | 1 | 17.4% | 160 | 3.7
Amazon reviews multilingual (polarity) McAuley & Leskovec (2013) | 960k | 24k | 2 | 6 | 35.2% | 27 | 5.0
MASSIVE intent FitzGerald et al. (2022) | 587k | 152k | 60 | 51 | 35.3% | 6 | 5.1

Table 3: List of datasets used for the classification evaluation.

To perform the most balanced evaluation possible we evaluate classification
accuracy on 5 different datasets that have drastically different training
size, number of languages, classification task, labels, and words per example
as reported in table 3. For example the MASSIVE FitzGerald et al. (2022)
intent classification dataset has very short sentences but a lot of languages
(51), whereas Yelp has very long examples and a lot of training data but a
single language: English.

All models are trained using TensorFlow 2.11 on a Google cloud machine using a
single NVidia V100. We split examples on whitespace for all languages except
for Chinese and Japanese for which we use Stanza Qi et al. (2020) whereas for
all other languages we split on whitespace. For brevity’s sake, we only report
the average accuracy across all languages for the Multilingual Amazon Reviews
and MASSIVE intent classification datasets.

| Architecture | Layers | Dim | Params
---|---|---|---|---
DPCNN Johnson & Zhang (2017) | CNN | 4 | 256 | 2.9M
Stacked LSTM Dee  | RNN | 4 | 256 | 1.6M
BERT-Mini Devlin et al. (2019) | Transformer | 4 | 256 | 3.3M

Table 4: Models architecture used throughout the paper evaluations. Parameters
count excludes the learned embedding size which vary per tokenizer.

We benchmark performance for the three most common model architectures:
Transformer (BERT Devlin et al. (2019)), CNN (DPCNN Johnson & Zhang (2017) )
and stacked-LSTM Dee  as reported in table 4. To make the evaluation fair and
consistent we keep the model parameters constants and only swap the tokenizers
between runs. Model key parameters are reported in table 4 and full details
are available in appendix B.

All models are trained with Adam optimizer with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, RNN and CNN models are trained with max learning rate of
$1e-3$ while BERT models use a max learning rate of 5e-4, with the exception
of fastText on multilingual datasets. We found that 5e-4 max learning was too
high and led to divergence so the max learning rate was set to 1e-4 instead
for fastText on MASSIVE and Amazon. For BERT models, we warm-up the learning
rate for 5k steps. All models are trained for 100k steps with a batch size 256
on every dataset, with cosine decay learning rate to 0. The average test
accuracy for each result over 3 trials is reported in Table 5.

#### Results

| Model | SentencePiece | BPE | fastText | Whitespace | RetVec-raw | RetVec-base | RetVec-large
---|---|---|---|---|---|---|---|---
AG News | RNN | 91.3% | 91.3% | 92.7% | 91.6% | 91.7% | 92.5% | 92.6%
CNN | 92.4% | 90.7% | 92.3% | 92.5% | 88.1% | 89.8% | 89.9%
BERT | 91.5% | 92.1% | 93.4% | 91.5% | 93.0% | 93.4% | 93.5%
AVG | 91.7% | 91.4% | 92.8% | 91.9% | 90.9% | 91.9% | 92.0%
DBPedia | RNN | 98.1% | 98.0% | 99.0% | 98.7% | 98.9% | 98.9% | 98.9%
CNN | 98.3% | 98.0% | 98.9% | 98.9% | 98.1% | 98.4% | 98.4%
BERT | 97.9% | 98.1% | 98.9% | 98.5% | 98.9% | 98.8% | 98.9%
AVG | 98.1% | 98.0% | 98.9% | 98.7% | 98.6% | 98.7% | 98.7%
Yelp P. | RNN | 93.3% | 93.4% | 94.7% | 93.9% | 94.0% | 94.6% | 94.6%
CNN | 93.8% | 93.0% | 93.9% | 94.4% | 92.3% | 93.1% | 93.1%
BERT | 91.3% | 91.6% | 93.5% | 91.6% | 93.1% | 92.7% | 92.7%
AVG | 92.8% | 92.7% | 94.0% | 93.3% | 93.2% | 93.5% | 93.5%
Multilingual Amazon P. | RNN | 90.7% | 90.3% | 87.0% | 93.2% | 93.0% | 93.3% | 93.4%
CNN | 90.8% | 89.2% | 84.4% | 93.2% | 90.7% | 91.3% | 91.5%
BERT | 87.3% | 87.2% | 86.6% | 91.8% | 92.6% | 92.3% | 92.4%
AVG | 89.6% | 88.9% | 86.0% | 92.8% | 92.1% | 92.3% | 92.4%
MASSIVE | RNN | 69.6% | 68.6% | 13.5% | 70.3% | 70.8% | 73.5% | 73.9%
CNN | 69.2% | 59.2% | 12.6% | 70.4% | 57.9% | 62.3% | 62.4%
BERT | 67.8% | 66.2% | 13.5% | 70.6% | 77.9% | 79.5% | 79.2%
AVG | 68.9% | 64.7% | 13.1% | 70.4% | 68.9% | 71.8% | 71.8%
Average | RNN | 88.6% | 88.3% | 77.4% | 89.5% | 89.7% | 90.6% | 90.7%
CNN | 88.9% | 86.0% | 76.4% | 89.9% | 85.4% | 87.0% | 87.1%
BERT | 87.2% | 87.0% | 78.0% | 88.8% | 91.1% | 91.4% | 91.3%
All | 88.2% | 87.1% | 77.2% | 89.4% | 88.7% | 89.6% | 89.7%

Table 5:  Classification results for various embedding schemes. Bold indicates
best results, underlined values are the 2nd best.

On average, RetVec-large is the best performing tokenizer by a small margin,
narrowly followed by RetVec-base. The difference between the two is marginal,
confirming that RetVec-base is the best performance/computation tradeoff. We
also observe that RetVec performs best when paired with the transformer (BERT)
and RNN (LSTM) architectures.

FastText performs the best on the English-only datasets but fails to perform
on the two multilingual datasets which explains its overall poor performance.
In the light of fastText bad results, we did a careful code review and
performed numerous additional tests which did yield any bugs. Given that we
are using the same code and models for all datasets and that fastText English-
only datasets results are inline with the ones reported in the original paper
Joulin et al. (2017), we currently believe that those bad results are due to
the fact that fastText vectors from different languages might not be playing
well together. However we didn’t find any mention of such an issue in the
literature.

##  6 Typo Resilience Evaluation

Typo | 0% | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100%
---|---|---|---|---|---|---|---|---|---|---|---
SentencePiece | 88.3% | 85.4% | 84.4% | 83.1% | 80.9% | 77.8% | 74.5% | 70.3% | 64.8% | 59.4% | 49.9%
BPE | 87.4% | 84.4% | 83.4% | 82.1% | 79.9% | 76.8% | 73.7% | 69.3% | 63.8% | 58.3% | 48.6%
fastText | 77.2% | 76.6% | 76.2% | 75.4% | 74.3% | 73.0% | 71.5% | 69.4% | 66.0% | 61.8% | 53.7%
Whitespace | 89.3% | 86.7% | 86.1% | 85.3% | 83.6% | 81.3% | 78.7% | 74.3% | 68.1% | 61.2% | 46.4%
RetVec-raw | 88.6% | 86.1% | 85.5% | 84.7% | 83.4% | 81.6% | 79.9% | 77.5% | 74.4% | 71.6% | 65.2%
RetVec-base | 89.5% | 87.9% | 87.6% | 87.3% | 86.7% | 85.9% | 85.3% | 84.2% | 83.1% | 82.3% | 79.8%
RetVec-large | 89.5% | 87.9% | 87.7% | 87.4% | 86.7% | 86.1% | 85.4% | 84.3% | 83.1% | 82.3% | 79.9%

Table 6:  Average classification accuracy for the models trained in Section 5
as the fraction of the words corrupted in each test sample increases. Bold
indicates best results, underlined values are the 2nd best.

In this section we evaluated the resilience of various embedders by evaluating
the best models trained in section 5 on increasingly corrupted test splits for
all 5 datasets.

#### Setup

We corrupt datasets by injecting a mix of typos that represent the common
types of typo made by humans: deletion, insertion, delimiters, missing or
wrong special characters, substitution, and transposition Hagen et al. (2017)
– see appendix D for the full list of augmentations used.

Typos are added at most once to each word, and the words to corrupt are
selected at random. We use a fixed seed to generate our datasets to make our
experiments reproducible and a generation script is available at [anonymized
for submission]. Each typo is applied with a block size of 1 or 2, which is
also randomly chosen.

Accordingly, when we report accuracy for a 20% typo rate it means that we
report the model accuracy for a dataset that was corrupted to have 20% of its
words with random typos.

#### Results

Overall, as reported in table 6, RetVec embedders’ resilience to typos are
consistently stronger than the other embeddings. As the amount of typos
increase, the gap in resilience widens with models using RetVec being able to
retain significantly more accuracy than models trained with other embedders.
RetVec-base and RetVec-large greater resilience compared to RetVec-raw
demonstrate the importance of the optional model to create robust classifiers.
Full results on typo resilience per dataset can be found in appendix 12.

##  7 Evaluation: Adversarial Attack

Vectorizer | SentencePiece | BPE | fastText | Whitespace | RetVec-raw | RetVec-base | RetVec-large
---|---|---|---|---|---|---|---
Base accuracy | 89.9% | 90.8% | 92.6% | 89.4% | 92.2% | 93.9% | 92.9%
Acc under Attack | 23.1% | 29.0% | 40.5% | 32.4% | 51.2% | 58.8% | 56.0%
Accuracy reduction | -66.8% | -61.8% | -52.1% | -57.0% | -41.0% | -35.1% | -36.9%
Attack success rate | 74.3% | 68.1% | 56.3% | 63.7% | 44.5% | 37.4% | 39.8%

Table 7: Adversarial resilience evaluation. Bold indicates best results,
underlined values are the 2nd best.

#### Setup

We used the Text Attack framework Morris et al. (2020) to evaluate the
resilience of various embedding schemes against character level adversarial
attacks, specifically: TextBuggerLi et al. (2019), WordBug Gao et al. (2018),
and Pruthi Pruthi et al. (2019). These attacks were carried out on 1000 random
examples drawn from the AGNews test dataset, using seven different BERT
models, one for each embedding scheme. In table 7, we report the average
classification results over the 1000 adversarial examples, with detailed
results in the appendix (table 20).

#### Results

The table 7 shows that RetVec models are significantly more resilient to
adversarial text attacks than other embedding schemes. In particular, the
attack success rate against the RetVec-base variant is 37.4%, which is roughly
half the attack success rate of SentencePiece (74.3%) and BPE (68.1%). The
best performing, non-RetVec, embedding scheme is fastText (56.3% attack
success rate), which makes sense supports our hypothesis that learned smooth
representations improve OOV performance. However, even the no-model RetVec-raw
(44.5%) variant significantly outperforms the other non-RetVec embedding
schemes with an 11.8% percentage point improvement over fastText.

Surprisingly, the attack success rate is higher against RetVec-large (39.8%)
than it is for RetVec-base (37.4%). As both models are trained in the same
manner, this seems to imply that the transformer architecture is less
resilient to text perturbations. This is unexpected as the dense architecture
of RetVec-base has a higher final training loss compared to RetVec-large (see
table 3). We aren’t aware of a good explanation for this phenomenon.

##  8 Experiments: In the wild evaluation

As a final evaluation, to ensure RetVec will work in practice against
adversarial content, we trained a version of a transformer model we use as
part of our spam email filtering system at [Anonymized]. As with previous
experiments, we kept everything constant beside swapping the embedder with
RetVec. Benchmarking on our internal production test dataset, we found out
that the model equipped with RetVec improved the recall at 0.80% false
positive rate by 0.51% compared to the same model using a SentencePiece
embedder. This result gives us confidence that RetVec is competitive for real-
world classification tasks purposes even in the face of adversarial content.

##  9 Experiments: Ablation study

In this section we report how key hyperparameters variations affect RetVec
models performance. Additional less impactful hyperparameters results are
reported in appendix C.

Note that as alluded earlier in section 3.3, the ablation study was performed
by pre-training RetVec on a single dataset round of augmented words (888M
words) instead of 10 for the final versions to save computation and reduce
carbon footprint. As a result, the accuracy gap between RetVec-base and
RetVec-large is significantly wider in those experiments than for the final
models.

| RetVec-base | RetVec-large
---|---|---
Activation | Loss | Accuracy | Loss | Accuracy
*tanh | 0.0054 | 92.30% | 0.0043 | 92.79%
sigmoid | 0.0065 | 92.30% | 0.0041 | 92.79%
relu1 | 0.0053 | 92.20% | 0.0063 | 92.65%
softsign | 0.0061 | 92.20% | 0.0057 | 92.58%

Table 8: Embedding activation. * denote the value selected for the final
model.

#### Embedding Activation

We report the effect of embedding activation function choice on downstream
classification performance in Table 8. We selected the most common activation
functions that will scale the embedding values between -1 and 1 or 0 to 1 in
the case of relu1. Overall we observed that hyperbolic tanh activation offer
the best performance for both RetVec-base and RetVec-large. It also the only
one to have a low loss values for both architectures.

#### Embedding Size

Embedding size | RetVec-base | RetVec-large
---|---|---
| Loss | Accuracy | Loss | Accuracy
64 | 0.0062 | 92.0% | - | -
100 | 0.0066 | 92.2% | - | -
128 | 0.0054 | 92.3% | 0.0048 | 92.6%
200 | 0.0052 | 92.3% | 0.0053 | 92.6%
*256 | 0.0052 | 92.2% | 0.0041 | 92.5%
300 | 0.0059 | 92.0% | 0.0052 | 92.5%
384 | 0.0052 | 92.2% | 0.0052 | 92.5%
512 | 0.0049 | 92.3% | 0.0146 | 92.5%

Table 9: Embedding size. * denote the value selected for the final model.

We train RetVec-base and RetVec-large with different embedding dimensions and
report the results in Table 9. Classification benchmark show little difference
in performance for embedding sizes between 128 up to 512 both for RetVec-base
and RetVec-large. We ended-up using a 256 embedding dim as it was the closest
power of two where the loss stop decrease for RetVec-base.

#### Max Input Character length

Avg |  Median |  Std |  p90 |  p95 |  p99 |  p99.9
---|---|---|---|---|---|---
8.4 |  7.9 |  4.6 |  13.0 |  15.0 |  20.8 |  36.1

Table 10: Word length statistics computed on all fastText words for 157
languages. p90 represents the 90th percentile, and p95 is the 95th percentile,
etc.

| RetVec-base | RetVec-large
---|---|---
Max Len | Loss | Accuracy | Loss | Accuracy
8 | 0.0198 | 92.3% | 0.0206 | 92.7%
10 | 0.0117 | 92.2% | 0.0105 | 92.7%
12 | 0.0076 | 92.1% | 0.0059 | 92.7%
14 | 0.006 | 92.2% | 0.0046 | 92.5%
*16 | 0.0054 | 92.3% | 0.0043 | 92.8%
24 | 0.0050 | 92.2% | 0.0048 | 92.7%
32 | 0.0053 | 92.2% | 0.0039 | 92.7%

Table 11: Max Input Character length. * denote the value selected for the
final model.

To select the optimal max word length we started by looking at fastText word
length distribution. As reported in table 10 more than 95% of the words are
less than 16 characters long and the median word length is 7.9 characters. We
did try to use more than 16 characters but this lead to downstream task
accuracy improvements as reported in table 11 despite those longer inputs
exhibiting a lower loss for RetVec-large. Accordingly we decided to use a max
length of 16 characters for both RetVec-base and RetVec-large as it offers the
best performance/compute cost (recall here that RetVec-large cost is quadratic
due to it’s transformer architecture).

##  10 Conclusion

In this paper we present RetVec a novel family of text embedding schemes that
combines a novel character encoding with an optional small model to convert
words into a 256 dimensional embedding. Extensive evaluation shows that models
trained using RetVec achieve state of the art classification accuracy and
exhibit strong resilience against adversarial text augmentation.

Understanding how to train the best pre-trained models possible on top of
RetVec is a clear extension of this work. Additionally, having more
informative and compact input potentially opens the door to create leaner
models, but also poses challenges as it prevents pre-training RetVec-based
models with off-the-shelves strategies such as word masking. We hope to
explore this direction in future work.

## References

  * (1) Deep Learning with Python.  https://www.manning.com/books/deep-learning-with-python.
  * (2) Hugging Face – The AI community building the future.  https://huggingface.co/.
  * Bet (2019) Better Language Models and Their Implications.  https://openai.com/blog/better-language-models/, February 2019.
  * Bojanowski et al. (2017) Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T.  Enriching Word Vectors with Subword Information, June 2017.
  * Chen & He (2020) Chen, X. and He, K.  Exploring Simple Siamese Representation Learning, November 2020.
  * Chowdhery et al. (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., and et al., M.  PaLM: Scaling Language Modeling with Pathways, October 2022\.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019.
  * FitzGerald et al. (2022) FitzGerald, J., Hench, C., Peris, C., Mackie, S., Rottmann, K., Sanchez, A., Nash, A., Urbach, L., Kakarala, V., Singh, R., Ranganath, S., Crist, L., Britan, M., Leeuwis, W., Tur, G., and Natarajan, P.  MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages, June 2022\.
  * Gao et al. (2018) Gao, J., Lanchantin, J., Soffa, M. L., and Qi, Y.  Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers, May 2018.
  * Grave et al. (2018) Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T.  Learning Word Vectors for 157 Languages, March 2018.
  * Hagen et al. (2017) Hagen, M., Potthast, M., Gohsen, M., Rathgeber, A., and Stein, B.  A Large-Scale Query Spelling Correction Corpus.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’17, pp. 1261–1264, New York, NY, USA, August 2017\. Association for Computing Machinery.  ISBN 978-1-4503-5022-8.  doi: 10.1145/3077136.3080749.
  * He et al. (2018) He, X., Zhou, Y., Zhou, Z., Bai, S., and Bai, X.  Triplet-Center Loss for Multi-View 3D Object Retrieval, March 2018\.
  * Hua et al. (2022) Hua, W., Dai, Z., Liu, H., and Le, Q. V.  Transformer Quality in Linear Time, June 2022.
  * Johnson & Zhang (2017) Johnson, R. and Zhang, T.  Deep Pyramid Convolutional Neural Networks for Text Categorization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 562–570, Vancouver, Canada, 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1052.
  * Joulin et al. (2017) Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T.  Bag of Tricks for Efficient Text Classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pp. 427–431, Valencia, Spain, April 2017\. Association for Computational Linguistics.
  * Kudo & Richardson (2018) Kudo, T. and Richardson, J.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, August 2018.
  * Li et al. (2019) Li, J., Ji, S., Du, T., Li, B., and Wang, T.  TextBugger: Generating Adversarial Text Against Real-world Applications.  In _Proceedings 2019 Network and Distributed System Security Symposium_ , 2019.  doi: 10.14722/ndss.2019.23138.
  * McAuley & Leskovec (2013) McAuley, J. and Leskovec, J.  Hidden factors and hidden topics: Understanding rating dimensions with review text.  In _Proceedings of the 7th ACM Conference on Recommender Systems_ , pp. 165–172, Hong Kong China, October 2013. ACM.  ISBN 978-1-4503-2409-0.  doi: 10.1145/2507157.2507163.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J.  Efficient Estimation of Word Representations in Vector Space, September 2013.
  * Morris et al. (2020) Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y.  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, October 2020.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C.  GloVe: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1532–1543, Doha, Qatar, October 2014. Association for Computational Linguistics.  doi: 10.3115/v1/D14-1162.
  * Pruthi et al. (2019) Pruthi, D., Dhingra, B., and Lipton, Z. C.  Combating Adversarial Misspellings with Robust Word Recognition, August 2019.
  * Qi et al. (2020) Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages, April 2020.
  * Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, July 2020.
  * Schroff et al. (2015) Schroff, F., Kalenichenko, D., and Philbin, J.  FaceNet: A Unified Embedding for Face Recognition and Clustering.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 815–823, June 2015.  doi: 10.1109/CVPR.2015.7298682.
  * Sennrich et al. (2016) Sennrich, R., Haddow, B., and Birch, A.  Neural Machine Translation of Rare Words with Subword Units, June 2016.
  * Sun et al. (2020a) Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L., Wang, Z., and Wei, Y.  Circle Loss: A Unified Perspective of Pair Similarity Optimization, June 2020a.
  * Sun et al. (2020b) Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L., Wang, Z., and Wei, Y.  Circle Loss: A Unified Perspective of Pair Similarity Optimization, June 2020b.
  * Wang et al. (2020) Wang, X., Han, X., Huang, W., Dong, D., and Scott, M. R.  Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning, March 2020.
  * (30) Xing, E. P., Ng, A. Y., Jordan, M. I., and Russell, S.  Distance metric learning, with application to clustering with side-information.
  * Xue et al. (2022) Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C.  ByT5: Towards a token-free future with pre-trained byte-to-byte models, March 2022.
  * Zhang et al. (2016) Zhang, X., Zhao, J., and LeCun, Y.  Character-level Convolutional Networks for Text Classification, April 2016.
  * Zhang & Saligrama (2016) Zhang, Z. and Saligrama, V.  Zero-Shot Learning via Joint Latent Similarity Embedding, August 2016.
  * Zhuang & Zuccon (2021) Zhuang, S. and Zuccon, G.  Dealing with Typos for BERT-based Passage Retrieval and Ranking, September 2021.

## Appendix

##  Appendix A Resilience

Full results for typo resilience across every dataset can be found in table
12.

Dataset | Vectorizer | 0% | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100%
---|---|---|---|---|---|---|---|---|---|---|---|---
Ag News | SentencePiece | 91.80% | 91.00% | 89.90% | 88.50% | 86.20% | 82.80% | 78.70% | 74.00% | 67.90% | 60.90% | 51.50%
Ag News | BPE | 91.60% | 90.50% | 89.10% | 87.60% | 84.80% | 80.60% | 76.10% | 70.20% | 63.80% | 56.20% | 46.30%
Ag News | fastText | 92.80% | 92.10% | 91.60% | 90.70% | 89.10% | 87.80% | 85.10% | 82.50% | 77.20% | 70.50% | 59.70%
Ag News | Whitespace-256 | 92.00% | 91.60% | 91.00% | 90.20% | 88.60% | 86.80% | 83.80% | 79.30% | 72.30% | 63.50% | 49.10%
Ag News | RetVec-raw | 91.10% | 90.40% | 89.50% | 88.40% | 87.10% | 85.40% | 83.80% | 81.40% | 78.30% | 74.90% | 69.00%
Ag News | RetVec-base | 92.10% | 91.60% | 91.30% | 90.80% | 90.10% | 89.40% | 89.10% | 87.90% | 87.00% | 85.80% | 84.10%
Ag News | RetVec-large | 92.20% | 91.60% | 91.40% | 91.00% | 90.10% | 90.00% | 89.10% | 88.00% | 87.10% | 85.90% | 84.60%
Multilingual Amazon P. | SentencePiece | 89.60% | 88.50% | 87.50% | 86.20% | 84.50% | 82.60% | 80.70% | 78.70% | 76.10% | 73.90% | 70.70%
Multilingual Amazon P. | BPE | 88.90% | 87.70% | 86.60% | 85.40% | 83.60% | 81.80% | 80.10% | 77.90% | 75.30% | 73.40% | 70.30%
Multilingual Amazon P. | fastText | 86.20% | 85.50% | 84.80% | 84.00% | 82.90% | 81.00% | 80.10% | 77.90% | 75.90% | 73.90% | 70.20%
Multilingual Amazon P. | Whitespace-256 | 92.70% | 92.10% | 91.60% | 91.10% | 90.20% | 89.10% | 87.90% | 86.20% | 83.70% | 80.90% | 75.00%
Multilingual Amazon P. | RetVec-raw | 91.90% | 91.20% | 90.30% | 89.50% | 88.10% | 86.50% | 84.80% | 83.10% | 80.70% | 78.50% | 74.60%
Multilingual Amazon P. | RetVec-base | 92.10% | 91.80% | 91.60% | 91.40% | 91.00% | 90.70% | 90.30% | 89.80% | 89.30% | 88.80% | 88.10%
Multilingual Amazon P. | RetVec-large | 92.30% | 92.10% | 91.80% | 91.60% | 91.20% | 91.00% | 90.60% | 90.20% | 89.80% | 89.30% | 88.80%
DBPedia | SentencePiece | 98.10% | 97.40% | 96.50% | 95.10% | 92.70% | 88.80% | 84.00% | 76.90% | 66.80% | 54.10% | 35.10%
DBPedia | BPE | 98.10% | 97.40% | 96.50% | 95.10% | 92.80% | 89.30% | 85.00% | 78.20% | 68.40% | 55.80% | 35.70%
DBPedia | fastText | 98.90% | 98.60% | 98.20% | 97.60% | 96.60% | 94.90% | 92.90% | 89.30% | 83.40% | 73.70% | 53.00%
DBPedia | Whitespace-256 | 98.70% | 98.20% | 97.50% | 96.60% | 94.80% | 92.10% | 88.30% | 81.90% | 71.70% | 57.10% | 29.90%
DBPedia | RetVec-raw | 98.40% | 98.10% | 97.60% | 96.90% | 95.90% | 94.50% | 92.80% | 90.40% | 87.00% | 82.50% | 75.10%
DBPedia | RetVec-base | 98.60% | 98.50% | 98.30% | 98.10% | 97.80% | 97.50% | 97.10% | 96.70% | 96.10% | 95.40% | 94.30%
DBPedia | RetVec-large | 98.60% | 98.50% | 98.30% | 98.10% | 97.80% | 97.40% | 97.00% | 96.60% | 95.90% | 95.10% | 93.80%
MASSIVE | SentencePiece | 69.00% | 58.50% | 58.40% | 57.80% | 55.80% | 52.70% | 50.70% | 46.70% | 41.80% | 40.90% | 29.60%
MASSIVE | BPE | 65.50% | 54.60% | 54.40% | 54.10% | 52.00% | 48.60% | 46.90% | 42.70% | 38.40% | 37.00% | 26.40%
MASSIVE | fastText | 14.20% | 13.10% | 13.50% | 12.70% | 12.30% | 11.80% | 11.40% | 11.40% | 10.60% | 10.50% | 10.00%
MASSIVE | Whitespace-256 | 70.00% | 58.90% | 58.80% | 58.40% | 55.80% | 51.70% | 49.20% | 43.90% | 37.10% | 34.80% | 17.30%
MASSIVE | RetVec-raw | 69.00% | 59.00% | 58.90% | 58.60% | 56.70% | 53.70% | 51.90% | 48.20% | 43.90% | 42.60% | 31.40%
MASSIVE | RetVec-base | 71.80% | 65.00% | 65.00% | 64.70% | 63.50% | 61.70% | 60.50% | 58.20% | 55.50% | 54.60% | 47.10%
MASSIVE | RetVec-large | 72.00% | 65.40% | 65.40% | 65.20% | 64.00% | 62.10% | 61.00% | 58.60% | 55.90% | 55.10% | 47.70%
Yelp P. | SentencePiece | 93.10% | 91.50% | 89.70% | 87.70% | 85.10% | 81.90% | 78.70% | 75.10% | 71.20% | 67.40% | 62.60%
Yelp P. | BPE | 93.00% | 91.70% | 90.20% | 88.50% | 86.20% | 83.50% | 80.50% | 77.20% | 73.10% | 69.00% | 64.20%
Yelp P. | fastText | 94.10% | 93.50% | 92.80% | 92.00% | 90.70% | 89.30% | 87.90% | 85.80% | 83.10% | 80.20% | 75.60%
Yelp P. | Whitespace-256 | 93.20% | 92.50% | 91.60% | 90.50% | 88.60% | 86.70% | 84.10% | 80.40% | 75.80% | 69.70% | 60.90%
Yelp P. | RetVec-raw | 92.60% | 91.90% | 91.10% | 90.20% | 89.00% | 87.70% | 86.20% | 84.50% | 81.90% | 79.50% | 75.60%
Yelp P. | RetVec-base | 92.80% | 92.40% | 92.00% | 91.50% | 90.90% | 90.10% | 89.60% | 88.50% | 87.60% | 86.70% | 85.20%
Yelp P. | RetVec-large | 92.60% | 92.10% | 91.70% | 91.20% | 90.50% | 89.80% | 89.00% | 88.10% | 87.00% | 86.00% | 84.50%

Table 12: Typo resilience on 0% to 100% word typo rate for each classification
dataset and embedding scheme. .

##  Appendix B Classification Models Details

This section provide the full parameters of the models used in the
classification evaluation section 5.

#### RNN

We used a stacked-LSTM architecture with the following hyperparamters

  * •

Dim: 256

  * •

Layers: 4

  * •

Dropout rate: 0.1

#### DPCNN

Used the exact architecture as described in Johnson & Zhang (2017)

  * •

Filters: 256

  * •

Layers: 6

  * •

Kernel size: 3

  * •

Final dropout: 0.5

#### BERT

Used the exact architecture as described in Devlin et al. (2019) BERT-Mini:

  * •

Layers: 4

  * •

Hidden dim: 256

  * •

Intermediate dim: 1024

  * •

Dropout rate: 0.1

##  Appendix C Ablation

###  C.1 Embedding Activation

We evaluate the effect of different activation functions on the embedding.

|  | RetVec-base |  | RetVec-large
---|---|---|---|---
Activation | Pre-training Loss | RNN | CNN | BERT | Average | Pre-training Loss | RNN | CNN | BERT | Average
*tanh | 0.0054 | 93.30% | 91.20% | 92.40% | 92.30% | 0.0043 | 93.46% | 91.95% | 92.97% | 92.79%
sigmoid | 0.0065 | 93.20% | 91.30% | 92.30% | 92.30% | 0.0041 | 93.65% | 92.04% | 92.68% | 92.79%
relu1 | 0.0053 | 93.00% | 91.40% | 92.30% | 92.20% | 0.0063 | 93.57% | 91.93% | 92.45% | 92.65%
softsign | 0.0061 | 93.20% | 91.10% | 92.30% | 92.20% | 0.0057 | 93.50% | 91.77% | 92.46% | 92.58%

Table 13: Full Embedding Activation.

###  C.2 Embedding Size

In this section, we evaluate the effect of different embedding sizes from 64
to 512 on RetVec-base and RetVec-large.

| Embedding size | RetVec-base |  | RetVec-large
---|---|---|---|---
| Pre-training Loss | RNN | CNN | BERT | Avg | Pre-training Loss | RNN | CNN | BERT | Avg
64 | 0.0062 | 93.00% | 91.00% | 92.20% | 92.00% | - | - | - | - | -
100 | 0.0066 | 93.10% | 91.20% | 92.40% | 92.20% | - | - | - | - | -
128 | 0.0054 | 93.30% | 91.20% | 92.40% | 92.30% | 0.0048 | 92.10% | 93.40% | 91.70% | 92.60%
200 | 0.0052 | 93.20% | 91.40% | 92.30% | 92.30% | 0.0053 | 92.00% | 93.40% | 92.00% | 92.60%
256 | 0.0052 | 93.30% | 91.30% | 92.10% | 92.20% | 0.0041 | 92.20% | 93.50% | 92.20% | 92.50%
300 | 0.0059 | 93.30% | 91.30% | 92.20% | 92.30% | 0.0052 | 92.00% | 93.70% | 92.10% | 92.50%
384 | 0.0052 | 93.30% | 91.10% | 92.30% | 92.20% | 0.0052 | 92.20% | 93.50% | 92.00% | 92.50%
512 | 0.0049 | 93.20% | 91.50% | 92.30% | 92.30% | 0.0146 | 92.00% | 93.60% | 91.50% | 92.50%
Table 14: Embedding size.

###  C.3 Max Character length

In this section, we analyze how maximum character length in the RetVec
optional model affects classification performance.

|  | RetVec-base |  | RetVec-large
---|---|---|---|---
Max Len | Pre-training Loss | RNN | CNN | BERT | Avg | Pre-training Loss | RNN | CNN | BERT | Avg
8 | 0.0198 | 93.2 | 91.3 | 92.3 | 92.3 | 0.0206 | 93.5 | 92 | 92.4 | 92.7
10 | 0.0117 | 93.1 | 91.2 | 92.2 | 92.2 | 0.0105 | 93.5 | 92.1 | 92.5 | 92.7
12 | 0.0076 | 92.2 | 90.9 | 92.3 | 92.1 | 0.0059 | 93.6 | 91.9 | 92.6 | 92.7
14 | 0.006 | 93.1 | 91.2 | 92.3 | 92.2 | 0.0046 | 93.5 | 91.9 | 92.5 | 92.5
*16 | 0.0054 | 93.3 | 91.2 | 92.4 | 92.3 | 0.0043 | 93.7 | 92 | 92.6 | 92.8
24 | 0.0050 | 93.3 | 91.2 | 92.2 | 92.2 | 0.0048 | 93.5 | 92 | 925 | 92.7
32 | 0.0053 | 93.2 | 91 | 92.3 | 92.2 | 0.0039 | 93.6 | 92.1 | 92.4 | 92.7

Table 15: Max Input Character length.

##  Appendix D Augmentation list

Below, we provide the full list of augmentations used to generate typo-
augmented words for the pre-training dataset.

  * •

Deletion

  * •

Substitution

    * –

Case substitution

    * –

n-grams based substitution for $n=3,4,5$

    * –

QWERTY keyboard typo substitution

    * –

Homoglyphs substitution

    * –

Random ASCII substitution

    * –

Random character from language alphabet substitution

  * •

Delimiters / Special Characters

    * –

Random punctuation substitution

    * –

Random separator substitution

    * –

Random BMP Unicode substitution

    * –

Random punctuation insertion

    * –

Random separator insertion

    * –

Suffix punctuation insertion

    * –

Prefix punctuation insertion

    * –

Random BMP Unicode insertion

    * –

Suffix emoji insertion

    * –

Prefix emoji insertion

  * •

Insertion

    * –

Repeated character

    * –

n-grams based prefix insertion for $n=3,4,5$

    * –

n-grams based suffix insertion for $n=3,4,5$

    * –

Random character from language alphabet insertion

  * •

Transposition

Max Input Length | 16
---|---
# of attention heads | 1
Attention kernel | $relu^{2}$
Attention type | Quadratic
FFN type | GAU
Activation | Swish
Norm. type | ScaleNorm
Absolute position emb. | ScaledSin
Relative position emb. | RoPE
Number of layers | 2
Hidden size | 32
Expansion rate | 1
Dropout rate | 0
Embedding activation | Tanh
Embedding dim | 256
Similarity dim | 256
Parameter count | 148,451
Table 16: RetVec-large model details. Max Input Length | 16
---|---
Activation | GeLU
Norm. type | Batch
Number of layers | 2
Hidden size | 32
Number of FC layers | 1
Dropout rate | 0
Embedding activation | Tanh
Embedding dim | 256
Similarity dim | 256
Parameter count | 201,536
Table 17: RetVec-base model details.

##  Appendix E RetVec Pre-training Details

We use the same training setup for pre-training RetVec-base and RetVec-large.
We use Multi-Similarity loss with $\alpha=4$, $\beta=80$, $\epsilon=0.1$ and
$\lambda=0.5$. Detailed optimizer parameters are reported in table 18. We
train for 8.8B tokens through ten rounds of the dataset as previously
mentioned.

Hyperparameter | RetVec Pre-training
---|---
Batch size | 1024
Adam $\epsilon$ | 1.00e-7
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.999
Peak learning rate | 0.001
End learning rate | 0.00001
Warm-up steps | 10000
Decay function | Cosine
Weight decay | 0
Table 18: Retvec pre-training hyperparameters.

Vectorizer | SentencePiece | BPE | FastText | Learned
---|---|---|---|---
Attack | TB | DW | P | Avg | TB | DW | P | Avg | TB | DW | P | Avg | TB | DW | P | Avg
Base accuracy | 89.9% | 89.9% | 89.9% | 89.9% | 90.8% | 90.8% | 90.8% | 90.8% | 92.6% | 92.6% | 92.6% | 92.6% | 89.4% | 89.4% | 89.4% | 89.4%
Acc under Attack | 4.5% | 2.5% | 62.4% | 23.1% | 8.3% | 0.5% | 78.1% | 29.0% | 22.9% | 18.1% | 80.5% | 40.5% | 10.4% | 3.3% | 83.6% | 32.4%
Accuracy reduction | -85.4% | -87.4% | -27.5% | -66.8% | -82.5% | -90.3% | -12.7% | -61.8% | -69.7% | -74.5% | -12.1% | -52.1% | -79.% | -86.1% | -5.8% | -57.%
Attack success rate | 94.99 | 97.22% | 30.59% | 74.3% | 90.7% | 99.4% | 14.% | 68.1% | 75.3% | 80.4% | 13.1% | 56.3% | 88.37% | 96.3% | 6.49% | 63.7%

Table 19: Detailed adversarial results.

Vectorizer | RetVec-raw | RetVec-base | RetVec-large
---|---|---|---
Attack | TB | DW | P | Avg | TB | DW | P | Avg | TB | DW | P | Avg
Base accuracy | 92.2% | 92.2% | 92.2% | 92.2% | 93.9% | 93.9% | 93.9% | 93.9% | 92.9% | 92.9% | 92.9% | 92.9%
Acc under Attack | 32.1% | 38.9% | 82.6% | 51.2% | 38.4% | 53.5% | 84.5% | 58.8% | 35.1% | 48.3% | 84.5% | 56.0%
Accuracy reduction | -60.1% | -53.3% | -9.6% | -41.0% | -55.5% | -40.4% | -9.4% | -35.1% | -57.8% | -44.6% | -8.4% | -36.9%
Attack success rate | 65.2% | 57.8% | 10.4% | 44.5% | 59.1% | 43.% | 10.% | 37.4% | 62.2% | 48.% | 9.04% | 39.8%

Table 20: Detailed adversarial results (continued).

Layers Count | Dim | FC layers | Pre-training Loss | RNN | CNN | BERT | Avg
---|---|---|---|---|---|---|---
1 | 128 | 1 | 0.0046 | 93.4% | 91.4% | 92.6% | 92.6%
1 | 64 | 1 | 0.0054 | 93.5% | 91.4% | 92.5% | 92.4%
1 | 96 | 1 | 0.005 | 93.2% | 91.6% | 92.3% | 92.4%
2 | 96 | 1 | 0.0041 | 93.3% | 91.3% | 92.4% | 92.3%
2 | 48 | 1 | 0.005 | 93.3% | 91.3% | 92.3% | 92.3%
2 | 64 | 1 | 0.0043 | 93.3% | 91.3% | 92.3% | 92.3%
2 | 128 | 1 | 0.0041 | 93.2% | 91.4% | 92.3% | 92.3%
2 | 32 | 3 | 0.0056 | 93.3% | 91.3% | 92.2% | 92.3%
3 | 128 | 1 | 0.0041 | 93.2% | 91.3% | 92.3% | 92.3%
2 | 32 | 2 | 0.0053 | 93.3% | 91.2% | 92.3% | 92.3%
3 | 96 | 1 | 0.0039 | 93.2% | 91.3% | 92.2% | 92.3%
2 | 32 | 0 | 0.0059 | 93.3% | 91.2% | 92.2% | 92.2%
1 | 48 | 1 | 0.0057 | 93.3% | 91.1% | 92.3% | 92.2%

Table 21: RetVec-base Model capacity ablation study.

Layers Count | Hidden Dim | Pre-training Loss | RNN | CNN | BERT | Average |
---|---|---|---|---|---|---|---
4 | 64 | 0.0049 | 93.6% | 92.1% | 92.8% | 92.8% |
2 | 32 | 0.0055 | 93.5% | 91.9% | 93.0% | 92.8% |
4 | 32 | 0.0041 | 93.7% | 92.0% | 92.7% | 92.8% |
4 | 48 | 0.0045 | 93.6% | 92.1% | 92.6% | 92.8% |
3 | 96 | 0.0065 | 93.5% | 92.2% | 92.5% | 92.7% |
3 | 48 | 0.0049 | 93.6% | 91.9% | 92.6% | 92.7% |
3 | 32 | 0.004 | 93.6% | 91.9% | 92.6% | 92.7% |
3 | 128 | 0.0087 | 93.6% | 92.1% | 92.4% | 92.7% |
4 | 96 | 0.007 | 93.7% | 91.8% | 92.6% | 92.7% |
3 | 64 | 0.0047 | 93.7% | 91.9% | 92.3% | 92.6% |
2 | 96 | 0.0061 | 93.7% | 91.8% | 92.4% | 92.6% |
4 | 128 | 0.0087 | 93.7% | 91.9% | 92.2% | 92.6% |
2 | 64 | 0.0055 | 93.5% | 91.9% | 92.3% | 92.6% |
2 | 128 | 0.0076 | 93.5% | 91.8% | 92.3% | 92.5% |
2 | 48 | 0.0214 | 93.3% | 91.3% | 92.5% | 92.4% |

Table 22: RetVec-large Model capacity ablation study.

##  Appendix F Ablation loss study

In this section we provide detailed ablation loss results.

$\alpha$ | $\beta$ | $\lambda$ | Pre-training Loss | RNN | CNN | BERT | Avg
---|---|---|---|---|---|---|---
4 | 80 | 0.5 | 0.005 | 93.30% | 91.20% | 92.40% | 92.30%
4 | 40 | 0.5 | 0.006 | 93.1% | 91.2% | 92.4% | 92.2%
4 | 20 | 0.5 | 0.006 | 93.2% | 91.2% | 92.1% | 92.2%
2 | 40 | 0.5 | 0.01 | 93.0% | 91.2% | 92.3% | 92.2%
2 | 80 | 0.5 | 0.011 | 93.1% | 91.2% | 92.2% | 92.2%
4 | 20 | 0.5 | 0.006 | 93.2% | 91.3% | 92.1% | 92.2%
2 | 20 | 0.5 | 0.009 | 92.9% | 91.1% | 92.3% | 92.1%
4 | 40 | 1 | 0.163 | 92.9% | 90.9% | 92.3% | 92.0%
2 | 20 | 1 | 0.145 | 92.8% | 91.0% | 92.2% | 92.0%
4 | 20 | 1 | 0.084 | 92.8% | 90.9% | 92.3% | 92.0%
4 | 80 | 1 | 0.18 | 92.8% | 90.6% | 92.3% | 91.9%
2 | 80 | 1 | 0.353 | 92.7% | 90.5% | 92.2% | 91.8%
2 | 40 | 1 | 0.312 | 92.4% | 90.6% | 92.2% | 91.8%

Table 23: RetVec-base Multi-Similarity Loss hyperparameters.

Scale factor $\gamma$ | Margin $m$ | Pre-training Loss | RNN | CNN | BERT | Avg
---|---|---|---|---|---|---
256 | 0.4 | 3.86 | 93.1% | 91.4% | 92.3% | 92.3%
128 | 0.4 | 2.05 | 93.1% | 91.2% | 92.4% | 92.2%
64 | 0.5 | 0.29 | 93.2% | 91.3% | 92.2% | 92.2%
512 | 0.5 | 2.15 | 93.1% | 91.2% | 92.3% | 92.2%
256 | 0.5 | 1.1 | 93.2% | 91.3% | 92.1% | 92.2%
128 | 0.5 | 0.54 | 93.1% | 91.2% | 92.2% | 92.2%
64 | 0.3 | 4.77 | 93.0% | 91.2% | 92.3% | 92.2%
128 | 0.3 | 6.68 | 93.2% | 91.1% | 92.3% | 92.2%
512 | 0.4 | 7.89 | 93.1% | 91.2% | 92.2% | 92.2%
512 | 0.3 | 24.51 | 93.0% | 91.0% | 92.4% | 92.2%
256 | 0 | 12.56 | 92.9% | 91.0% | 92.5% | 92.1%
64 | 0.4 | 1.28 | 92.9% | 91.2% | 92.2% | 92.1%

Table 24: RetVec-base Circle Loss Hyperparameters.

$\alpha$ | $\beta$ | $\lambda$ | Pre-training Loss | RNN | CNN | BERT | Avg
---|---|---|---|---|---|---|---
4 | 80 | 0.5 | 0.005 | 93.5% | 91.9% | 93.0% | 92.8%
1 | 80 | 0.5 | 0.014 | 93.6% | 92.0% | 92.6% | 92.7%
2 | 80 | 0.5 | 0.01 | 93.6% | 92.0% | 92.5% | 92.7%
2 | 40 | 0.5 | 0.007 | 93.6% | 92.0% | 92.4% | 92.7%
1 | 40 | 0.5 | 0.017 | 93.5% | 92.1% | 92.4% | 92.7%
4 | 40 | 0.5 | 0.005 | 93.5% | 91.9% | 92.6% | 92.7%
2 | 20 | 0.5 | 0.008 | 93.4% | 92.0% | 92.5% | 92.6%
1 | 20 | 0.5 | 0.012 | 93.7% | 91.8% | 92.5% | 92.6%
4 | 20 | 0.5 | 0.004 | 93.6% | 91.8% | 92.4% | 92.6%
2 | 40 | 1 | 0.293 | 93.3% | 91.8% | 92.6% | 92.6%
2 | 20 | 1 | 0.123 | 93.4% | 91.8% | 92.4% | 92.5%
1 | 40 | 1 | 0.585 | 93.2% | 91.7% | 92.6% | 92.5%
1 | 20 | 1 | 0.25 | 93.4% | 91.6% | 92.5% | 92.5%
4 | 40 | 1 | 0.158 | 93.4% | 91.5% | 92.5% | 92.4%
2 | 80 | 1 | 0.352 | 93.3% | 91.6% | 92.4% | 92.4%
4 | 80 | 1 | 0.179 | 93.3% | 91.4% | 92.6% | 92.4%
4 | 20 | 1 | 0.068 | 93.3% | 91.4% | 92.5% | 92.4%
1 | 80 | 1 | 0.699 | 93.2% | 91.3% | 92.5% | 92.3%

Table 25: RetVec-large Multi-Similarity Loss hyperparameters.

Scale factor $\gamma$ | Relaxation factor $m$ | Pre-training Loss | RNN | CNN | BERT | Avg
---|---|---|---|---|---|---
64 | 0.5 | 0.19 | 93.7% | 92.0% | 92.6% | 92.8%
128 | 0.5 | 0.36 | 93.5% | 92.0% | 92.7% | 92.7%
256 | 0.4 | 2.47 | 93.7% | 92.0% | 92.5% | 92.7%
64 | 0.3 | 3.39 | 93.5% | 92.1% | 92.6% | 92.7%
128 | 0.2 | 9.83 | 93.5% | 92.0% | 92.6% | 92.7%
64 | 0.4 | 0.73 | 93.6% | 92.0% | 92.4% | 92.7%
128 | 0.4 | 1.51 | 93.6% | 92.0% | 92.5% | 92.7%
512 | 0.5 | 1.4 | 93.7% | 91.9% | 92.4% | 92.6%
64 | 0.2 | 7.34 | 93.5% | 91.9% | 92.5% | 92.6%
512 | 0.4 | 4.79 | 93.5% | 91.8% | 92.6% | 92.6%
256 | 0 | 17.16 | 93.3% | 91.9% | 92.7% | 92.6%
256 | 0.3 | 7.25 | 93.6% | 91.8% | 92.4% | 92.6%
128 | 0.3 | 3.88 | 93.5% | 91.8% | 92.5% | 92.6%
512 | 0.2 | 33.4 | 93.4% | 91.9% | 92.5% | 92.6%
256 | 0.5 | 0.78 | 93.7% | 91.7% | 92.4% | 92.6%
512 | 0.3 | 51.21 | 93.4% | 91.5% | 92.5% | 92.5%

Table 26:  RetVec-large Circle loss hyperparameters.

Objectives | RetVec-base | RetVec-large
---|---|---
CLS Objectives | Seq Objectives | Loss | RNN | CNN | BERT | Average | Loss | RNN | CNN | BERT | Average
N/A | Similarity, Augmentation Type Detection | 0.00835 | 93.20% | 91.10% | 92.40% | 92.20% | 0.00643 | 93.70% | 91.80% | 92.40% | 92.60%
N/A | Similarity, Augmentation Position Prediction | 0.00866 | 93.20% | 91.10% | 92.10% | 92.10% | 0.00809 | 93.60% | 91.80% | 92.50% | 92.60%
N/A | Similarity, Denoising | 0.01132 | 93.00% | 90.80% | 92.30% | 92.10% | 0.00598 | 93.50% | 91.70% | 92.70% | 92.60%
Similarity | Augmentation Type Detection | - | - | - | - | - | 0.00811 | 91.70% | 89.40% | 91.40% | 90.80%
Similarity | Augmentation Position Prediction | - | - | - | - | - | 0.00634 | 91.80% | 89.70% | 91.90% | 91.10%
Similarity | Denoising | - | - | - | - | - | 0.00706 | 87.50% | 86.10% | 85.90% | 86.50%

Table 27:  RetVec-large training objectives.