# Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust
with Little Cost

Lihu Chen1, Gaël Varoquaux2, Fabian M. Suchanek1
1 LTCI & Télécom Paris & Institut Polytechnique de Paris, France
2 Soda, Inria Saclay & CEA & Université Paris-Saclay, France
{lihu.chen<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

State-of-the-art NLP systems represent inputs with word embeddings, but these
are brittle when faced with Out-of-Vocabulary (OOV) words. To address this
issue, we follow the principle of mimick-like models to generate vectors for
unseen words, by learning the behavior of pre-trained embeddings using only
the surface form of words. We present a simple contrastive learning framework,
LOVE, which extends the word representation of an existing pre-trained
language model (such as BERT), and makes it robust to OOV with few additional
parameters. Extensive evaluations demonstrate that our lightweight model
achieves similar or even better performances than prior competitors, both on
original datasets and on corrupted variants. Moreover, it can be used in a
plug-and-play fashion with FastText and BERT, where it significantly improves
their robustness.

##  1 Introduction

Word embeddings represent words as vectors Mikolov et al. (2013a, b);
Pennington et al. (2014). They have been instrumental in neural network
approaches that brought impressive performance gains to many natural language
processing (NLP) tasks. These approaches use a fixed-size vocabulary. Thus
they can deal only with words that have been seen during training. While this
works well on many benchmark datasets, real-word corpora are typically much
noisier and contain Out-of-Vocabulary (OOV) words, i.e., rare words, domain-
specific words, slang words, and words with typos, which have not been seen
during training. Model performance deteriorates a lot with unseen words, and
minor character perturbations can flip the prediction of a model Liang et al.
(2018); Belinkov and Bisk (2018); Sun et al. (2020); Jin et al. (2020). Simple
experiments (Figure 1) show that the addition of typos to datasets degrades
the performance for text classification models considerably.

Figure 1: Performances of existing word embeddings as we gradually add typos
to the datasets. Using our model, LOVE, to produce vectors for OOV words makes
the models more robust.

To alleviate this problem, pioneering work pre-trained word embeddings with
morphological features (sub-word tokens) on large-scale datasets Wieting et
al. (2016); Bojanowski et al. (2017); Heinzerling and Strube (2017); Zhang et
al. (2019). One of the most prominent works in this direction is FastText
Bojanowski et al. (2017), which incorporates character n-grams into the skip-
gram model. With FastText, vectors of unseen words can be imputed by summing
up the n-gram vectors. However, these subword-level models come with great
costs: the requirements of pre-training from scratch and high memory
footprint. Hence, several simpler approaches have been developed, e.g., MIMICK
Pinter et al. (2017), BoS Zhao et al. (2018) and KVQ-FH Sasaki et al. (2019).
These use only the surface form of words to generate vectors for unseen words,
through learning from pre-trained embeddings.

Although MIMICK-like models can efficiently reduce the parameters of pre-
trained representations and alleviate the OOV problem, two main challenges
remain. First, the models remain bound in the trade-off between complexity and
performance: The original MIMICK is lightweight but does not produce high-
quality word vectors consistently. BoS and KVQ-FH obtain better word
representations but need more parameters. Second, these models cannot be used
with existing pre-trained language models such as BERT. It is these models,
however, to which we owe so much progress in the domain Peters et al. (2018);
Devlin et al. (2019); Yang et al. (2019); Liu et al. (2020). To date, these
high-performant models are still fragile when dealing with rare words Schick
and Schütze (2020), misspellings Sun et al. (2020) and domain-specific words
El Boukkouri et al. (2020).

Figure 2: Our lightweight OOV model, LOVE, learns the behavior of pre-trained
embeddings (e.g., FastText and BERT), and is then able to impute vectors for
unseen words. LOVE can enhance the robustness of existing word representations
in a plug-and-play fashion.

We address these two challenges head-on: we design a new contrastive learning
framework to learn the behavior of pre-trained embeddings, dubbed LOVE,
Learning Out-of-Vocabulary Embeddings. Our model builds upon a memory-saving
mixed input of character and subwords instead of n-gram characters. It encodes
this input by a lightweight Positional Attention Module. During training, LOVE
uses novel types of data augmentation and hard negative generation. The model
is then able to produce high-quality word representations that are robust to
character perturbations, while consuming only a fraction of the cost of
existing models. For instance, LOVE with 6.5M parameters can obtain similar
representations as the original FastText model with more than 900M parameters.
What is more, our model can be used in a plug-and-play fashion to robustify
existing language models. We find that using LOVE to produce vectors for
unseen words improves the performance of FastText and BERT by around 1.4-6.8
percentage points on noisy text – without hampering their original
capabilities (As shown in Figure 2).

In the following, Section 2 discusses related work, Section 3 introduces
preliminaries, Section 4 presents our approach, Section 5 shows our
experiments, and Section 6 concludes. The appendix contains additional
experiments and analyses. Our code and data is available at
https://github.com/tigerchen52/LOVE

##  2 Related Work

###  2.1 Character-level Embeddings

To address OOV problems, some approaches inject character-level features into
word embeddings during the pre-training Wieting et al. (2016); Cao and Rei
(2016); Bojanowski et al. (2017); Heinzerling and Strube (2017); Kim et al.
(2018); Li et al. (2018); Üstün et al. (2018); Piktus et al. (2019); Zhu et
al. (2019); Zhang et al. (2019); Hu et al. (2019). One drawback of these
methods is that they need to pre-train on a large-scale corpus from scratch.
Therefore, simpler models have been developed, which directly mimic the well-
trained word embeddings to impute vectors for OOV words. Some of these methods
use only the surface form of words to generate embeddings for unseen words
Pinter et al. (2017); Zhao et al. (2018); Sasaki et al. (2019); Fukuda et al.
(2020); Jinman et al. (2020), while others use both surface and contextual
information to create OOV vectors Schick and Schütze (2019a, b). In both
cases, the models need an excessive number of parameters. FastText, e.g., uses
~2 million n-gram characters to impute vectors for unseen words.

###  2.2 Pre-trained Language Models

Currently, the state-of-the-art word representations are pre-trained language
models, such as ELMo Peters et al. (2018), BERT Devlin et al. (2019) and XLnet
Yang et al. (2019), which adopt subwords to avoid OOV problems. However, BERT
is brittle when faced with rare words Schick and Schütze (2020) and
misspellings Sun et al. (2020). To make BERT more robust, CharacterBERT El
Boukkouri et al. (2020) and CharBERT Ma et al. (2020) infuse character-level
features into BERT and pre-train the variant from scratch. This method can
significantly improve the performance and robustness of BERT, but requires
pre-training an adapted transformer on a large amount of data. Another work on
combating spelling mistakes recommends placing a word corrector before
downstream models Pruthi et al. (2019), which is effective and reusable. The
main weakness of this method is that an error generated by the word corrector
propagates to downstream tasks. For example, converting “aleph” to “alpha” may
break the meaning of a mathematical statement. And indeed, using the word
corrector consistently leads to a drop (0.5-2.0 percentage points) in BERT’s
performance on the SST dataset Socher et al. (2013).

###  2.3 Contrastive Learning

The origin of contrastive learning can be traced back to the work by Becker
and Hinton (1992) and Bromley et al. (1993). This method has achieved
outstanding success in self-supervised representation learning for images Oord
et al. (2018); Hjelm et al. (2018); He et al. (2020); Chen et al. (2020);
Grill et al. (2020). The contrastive learning framework learns representations
from unlabeled data by pulling positive pairs together and pushing negative
pairs apart. For training, the positive pairs are often obtained by taking two
randomly augmented versions of the same sample and treating the other
augmented examples within a mini-batch as negative examples Chen et al. (2017,
2020). The most widely used loss is the infoNCE loss (or contrastive loss)
Hjelm et al. (2018); Logeswaran and Lee (2018); Chen et al. (2020); He et al.
(2020). Although many approaches adopt contrastive learning to represent
sentences Giorgi et al. (2020); Wu et al. (2020); Gao et al. (2021), it has so
far not been applied to word representations.

| Input | Encoder | Loss
---|---|---|---
MIMICK Pinter et al. (2017) |  character sequence {s,p,e,l,l} | RNNs | $\mathcal{L}_{\mathrm{dis}}$
BoS Zhao et al. (2018) |  n-gram subword {spe,pel,ell} | SUM | $\mathcal{L}_{\mathrm{dis}}$
KVQ-FH Sasaki et al. (2019) |  adapted n-gram subword {spe,pel,ell} | Attention | $\mathcal{L}_{\mathrm{dis}}$
Table 1: Details of different mimick-like models, with the word spell as an
example.

##  3 Preliminaries

###  3.1 Mimick-like Model

Given pre-trained word embeddings, and given an OOV word, the core idea of
MIMICK Pinter et al. (2017) is to impute an embedding for the OOV word using
the surface form of the word, so as to mimic the behavior of the known
embeddings. BoS Zhao et al. (2018), KVQ-FH Sasaki et al. (2019), Robust
Backed-off Estimation Fukuda et al. (2020), and PBoS Jinman et al. (2020) work
similarly, and we refer to them as mimick-like models.

Formally, we have a fixed-size vocabulary set $\mathcal{V}$, with an embedding
matrix $\mathbf{W}\in\mathbb{R}^{|\mathcal{V}|\times m}$, in which each row is
a word vector $\mathbf{u}_{w}\in\mathbb{R}^{m}$ for the word $w$. A mimick-
like model aims to impute a vector $\mathbf{v}_{w}$ for an arbitrary word
$w\not\in\mathcal{V}$. The training objective of mimick-like models is to
minimize the expected distance between $\mathbf{u}_{w}$ and $\mathbf{v}_{w}$
pairs:

$\mathcal{L}_{\mathrm{dis}}=\frac{1}{|\mathcal{V}|}\sum_{w\in\mathcal{V}}\psi(\mathbf{u}_{w},\mathbf{v}_{w})$
(1)

Here, $\psi(\cdot)$ is a distance function, e.g., the Euclidean distance
$\psi=\left\|\mathbf{u}_{w}-\mathbf{v}_{w}\right\|^{2}_{2}$ or the cosine
distance $\psi=1-\cos(\mathbf{u}_{w},\mathbf{v}_{w})$. The vector
$\mathbf{v}_{w}$ is generated by the following equation:

$\mathbf{v}_{w}=\phi(\zeta(w)),\text{ for }w\in\mathcal{V}\text{ or
}w\notin\mathcal{V}$ (2)

Here, $\zeta(\cdot)$ is a function that maps $w$ to a list of subunits based
on the surface form of the word (e.g., a character or subword sequence). After
that, the sequence is fed into the function $\phi(\cdot)$ to produce vectors,
and the inside structure can be CNNs, RNNs, or a simple summation function.
After training, the model can impute vectors for arbitrary words. Table 1
shows some features of three mimick-like models. color=yellow]Lihu done

###  3.2 Contrastive Learning

Contrastive learning methods have achieved significant success for image
representation Oord et al. (2018); Chen et al. (2020). The core idea of these
methods is to encourage learned representations for positive pairs to be
close, while pushing representations from sampled negative pairs apart. The
widely used contrastive loss Hjelm et al. (2018); Logeswaran and Lee (2018);
Chen et al. (2020); He et al. (2020) is defined as:

$\displaystyle\ell_{cl}$
$\displaystyle=-\log\frac{e^{\text{sim}(\mathbf{u}_{i}^{\mathsf{T}}\mathbf{u}^{+})/\tau}}{e^{\text{sim}(\mathbf{u}_{i}^{\mathsf{T}}\mathbf{u}^{+})/\tau}+\sum
e^{\text{sim}(\mathbf{u}_{i}^{\mathsf{T}}\mathbf{u}^{-})/\tau}}$ (3)

Here, $\tau$ is a temperature parameter, $\text{sim}(\cdot)$ is a similarity
function such as cosine similarity, and $(u_{i},u^{+})$, $(u_{i},u^{-})$ are
positive pairs and negative pairs, respectively (assuming that all vectors are
normalized). During training, positive pairs are usually obtained by
augmentation for the same sample, and negative examples are the other samples
in the mini-batch. This process learns representations that are invariant
against noisy factors to some extent.

Figure 3: The framework of LOVE with an example of the word misspelling.

##  4 Our Approach: LOVE

LOVE (Learning Out-of-Vocabulary Embeddings) draws on the principles of
contrastive learning to maximize the similarity between target and generated
vectors, and to push apart negative pairs. An overview of our framework is
shown in Figure 3. It is inspired by work in visual representation learning
Chen et al. (2020), but differs in that one of the positive pairs is obtained
from pre-trained embeddings instead of using two augmented versions. We adopt
five novel types of word-level augmentations and a lightweight Positional
Attention Module in this framework. Moreover, we find that adding hard
negatives during training can effectively yield better representations. We
removed the nonlinear projection head after the encoder layer, because its
improvements are specific to the representation quality in the visual field.
Furthermore, our approach is not an unsupervised contrastive learning
framework, but a supervised learning approach.

Our framework takes a word from the original vocabulary and uses data
augmentation to produce a corruption of it. For example, "misspelling" becomes
"mispelling" after dropping one letter "s". Next, we obtain a target vector
from the pre-trained embeddings for the original word, and we generate a
vector for the corrupted word. These two vectors are a pair of positive
samples, and we maximize the similarity between them while making the distance
of negative pairs (other samples in the same mini-batch) as large as possible.
As mentioned before, we use the contrastive loss as an objective function (Eq
3). There are five key ingredients in the framework that we will detail in the
following (similar to the ones in Table 1): the Input Method, the Encoder, the
Loss Function, our Data Augmentation, and the choice of Hard Negatives.

###  4.1 Input Method

Our goal is to use the surface form to impute vectors for words. The question
is thus how to design the function $\zeta(\cdot)$ mentioned in Section 3.1 to
represent each input word. MIMICK Pinter et al. (2017) straightforwardly uses
the character sequence (see Table 1). This, however, loses the information of
morphemes, i.e., sequences of characters that together contribute a meaning.
Hence, FastText Bojanowski et al. (2017) adopts character n-grams. Such
n-grams, however, are highly redundant. For example, if we use substrings of
length 3 to 5 to represent the word misspelling, we obtain a list with 24
n-gram characters – while only the substrings {mis, spell, ing} are the three
crucial units to understand the word. Hence, like BERT, we use WordPiece Wu et
al. (2016) with a vocabulary size of around 30000 to obtain meaningful
subwords of the input word. For the word misspelling, this yields {miss,
##pel, ##ling }. However, if we just swap two letters (as by a typo), then the
sequence becomes completely different: {mi, ##sp, ##sell, ##ing }. Therefore,
we use both the character sequence and subwords (Figure A1).

We shrink our vocabulary by stemming all words and keeping only the base form
of each word, and by removing words with numerals. This decreases the size of
vocabulary from 30 000 to 21 257 without degrading performance too much
(Section A.1).

###  4.2 Encoder

Let us now design the function $\phi(\cdot)$ mentioned in Section 3.1. We are
looking for a function that can encode both local features and global
features. Local features are character n-grams, which provide robustness
against minor variations such as character swaps or omissions. Global features
combine local features regardless of their distance. For the word misspelling,
a pattern of prefix and suffix mis+ing can be obtained by combining the local
information at the beginning and the end of the word. Conventional CNNs, RNNs,
and self-attention cannot extract such local and global information at the
same time. Therefore, we design a new Positional Attention Module. Suppose we
have an aforementioned mixed input sequence and a corresponding embedding
matrix $\mathbf{V}\in\mathbb{R}^{|\mathcal{V}|\times d}$ where $d$ is the
dimension of vectors. Then the input can be represented by a list of vectors:
$\mathbf{X}=\\{\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{n}\\}\in\mathbb{R}^{n\times
d}$ where $n$ is the length of the input. To extract local information, we
first adopt positional attention to obtain n-gram features, and then feed them
into a conventional self-attention layer to combine them in a global way. This
can be written as:

$\displaystyle\mathbf{\bar{X}}$
$\displaystyle=\mathrm{SA}(\mathrm{PA}(\mathbf{X}))\,\mathbf{W}^{O}$ (4)

Here, SA is a standard multi-head self-attention and PA is a positional
attention. $\mathbf{W}^{O}\in\mathbb{R}^{d_{V}\times d_{O}}$ is a trainable
parameter matrix, where $d_{V}$ are the dimensions of values in SA and PA, and
$d_{O}$ is that of $\mathbf{\bar{X}}$. As for the Positional Attention, we
adopt absolute sinusoidal embeddings Vaswani et al. (2017) to compute
positional correlations:

$\displaystyle\mathrm{PA}(\mathbf{X})=\mathrm{Softmax}\biggl{(}\frac{\mathbf{P}\mathbf{P}^{\mathsf{T}}}{\sqrt{d}}\biggr{)}(\mathbf{X\,W}^{V})$
(5)

Here, $\mathbf{P}\in\mathbb{R}^{n\times d}$ are the position embeddings, and
$\mathbf{W}^{V}\in\mathbb{R}^{d\times d_{V}}$ are the corresponding
parameters. More details about the encoder are in Appendix C.4.

###  4.3 Loss Function

In this section, we focus on the loss function $\mathcal{L}(\cdot)$. Mimick-
like models often adopt the mean squared error (MSE), which tries to give
words with the same surface forms similar embeddings. However, the MSE only
pulls positive word pairs closer, and does not push negative word pairs apart.
Therefore, we use the contrastive loss instead (Equation 3). Wang and Isola
(2020) found that the contrastive loss optimizes two key properties: Alignment
and Uniformity. The Alignment describes the expected distance (closeness)
between positive pairs:

$\ell_{\text{align}}\triangleq\mathop{\mathbb{E}}\limits_{(x,y)\sim
p_{\text{pos}}}{\psi(\mathbf{u}_{x},\mathbf{u}_{y})}$ (6)

Here, $p_{\text{pos}}$ is the distribution of positive pairs. The Uniformity
measures whether the learned representations are uniformly distributed in the
hypersphere:

$\ell_{\text{uniform}}\triangleq\log\mathop{\mathbb{E}}\limits_{(x,y)\overset{i.i.d.}{\sim}\
p_{\text{data}}}{e^{-t\cdot\psi(\mathbf{u}_{x},\mathbf{u}_{y})}}$ (7)

Here, $p_{\text{data}}$ is the data distribution and $t>0$ is a parameter. The
two properties are consistent with our expected word representations: positive
word pairs should be kept close and negative word pairs should be far from
each other, finally scattered over the hypersphere.

###  4.4 Data Augmentation and Hard Negatives

Our positive word pairs are generated by data augmentation, which can increase
the amount of training samples by using existing data. We use various
strategies (Figure 4) to increase the diversity of our training samples: (1)
Swap two adjacent characters, (2) Drop a character, (3) Insert a new
character, (4) Replace a character according to keyboard distance, (5) Replace
the original word by a synonymous word. The first four augmentations are
originally designed to protect against adversarial attacks Pruthi et al.
(2019). We add the synonym replacement strategy to keep semantically similar
words close in the embedding space – something that cannot be achieved by the
surface form alone. Specifically, a set of synonyms is obtained by retrieving
the nearest neighbors from pre-trained embeddings like FastText.

Figure 4: Illustrations of different augmentations for the word misspelling.

Negative word pairs are usually chosen randomly from the mini-batch. However,
we train our model to be specifically resilient to _hard negatives (or
difficult negatives)_ , i.e., words with similar surface forms but different
meanings (e.g., misspelling and dispelling). To this end, we add a certain
number of hard negative samples (currently 3 of them) to the mini-batch, by
selecting word pairs that are not synonyms and have a small edit distance.

###  4.5 Mimicking Dynamical Embeddings

Pre-trained Language Models (e.g., ELMo Peters et al. (2018) and BERT Devlin
et al. (2019)) dynamically generate word representations based on specific
contexts, which cannot be mimicked directly. To this end, we have two options:
We can either learn the behavior of the _input embeddings_ in BERT before the
multi-layer attentions or mimic the static _distilled embeddings_ Bommasani et
al. (2020); Gupta and Jaggi (2021).

We use the BERT as an example to explain these two methods. Suppose we have a
subword sequence after applying WordPiece to a sentence:
$W=\\{w_{1},w_{2},...,w_{n}\\}$. For the subword sequence $W$, BERT first
represents it as a list of subword embeddings:
$\mathbf{E}^{in}=\\{\mathbf{e}^{sub}_{1},\mathbf{e}^{sub}_{2},...,\mathbf{e}^{sub}_{n}\\}$.
We refer to this static representation as the Input Embedding of BERT, and we
can use our model to mimic the behavior of this part. We call this method
_mimicking input embeddings_. For ease of implementation, we learn only from
the words that are not separated into pieces. After that step, BERT applies a
multi-layer multi-head attention to the input embeddings $E^{in}$, which
yields a contextual representation for each subword:
$\mathbf{E}^{out}=\\{\mathbf{e}^{out}_{1},\mathbf{e}^{out}_{2},...,\mathbf{e}^{out}_{n}\\}$.
However, these contextual representations vary according to the input sentence
and we cannot learn from them directly. Instead, we choose to mimic the
distilled static embeddings from BERT, which are obtained by pooling (max or
average) the contextual embeddings of the word in different sentences. We call
this method _mimicking distilled embeddings_. The latter allows for better
word representations, while the former does not require training on a large-
scale corpus. Our empirical studies show that mimicking distilled embeddings
performs only marginally better. Therefore, we decided to rather learn the
input embeddings of BERT, which is simple yet effective

###  4.6 Plug and Play

One of the key advantages of our model is that it can be used as a plug-in for
other models. For models with static word embeddings like FastText, one can
simply use our model to generate vectors for unseen words. For models with
dynamic word embeddings like BERT, if a single word is tokenized into several
parts, e.g. misspelling = {miss, ##pel, ##ling }, we regard it as an OOV word.
Then, we replace the embeddings of the subwords by a single embedding produced
by our model before the attention layer. Our final enhanced BERT model has 768
dimensions and 16M parameters. Note that the BERT-base model has ~110M
parameters and its distilled one has ~550M parameters.

| parameters | Word Similarity | Word Cluster | Avg
---|---|---|---|---
| embedding | others | RareWord | SimLex | MTurk | MEN | WordSim | SimVerb | AP | BLESS |
FastText Bojanowski et al. (2017) | 969M | - | 48.1 | 30.4 | 66.9 | 78.1 | 68.2 | 25.7 | 58.0 | 71.5 | 55.9
MIMICK Pinter et al. (2017) | 9M | 517K | 27.1 | 15.9 | 32.5 | 36.5 | 15.0 | 7.5 | 59.3 | 72.0 | 33.2
BoS Zhao et al. (2018) | 500M | - | 44.2 | 27.4 | 55.8 | 65.5 | 53.8 | 22.1 | 41.8 | 39.0 | 43.7
KVQ-FH Sasaki et al. (2019) | 12M | - | 42.4 | 20.4 | 55.2 | 63.4 | 53.1 | 16.4 | 39.1 | 42.5 | 41.6
LOVE | 6.3M | 200K | 42.2 | 35.0 | 62.0 | 68.8 | 55.1 | 29.4 | 53.2 | 51.5 | 49.7
Table 2: Performance on the intrinsic tasks, measured as Spearman’s $\rho$ and purity for word similarity and clustering. Best performance among the mimick-like models in bold, second-best underlined. | parameters | SST2 | MR | CoNLL-03 | BC2GM | Avg
---|---|---|---|---|---|---
| embedding | others | original | +typo | original | +typo | original | +typo | original | +typo |
FastText Bojanowski et al. (2017) | 969M | - | 82.3 | 60.5 | 73.3 | 62.2 | 86.4 | 66.3 | 71.8 | 53.4 | 69.5
Edit Distance | 969M | - | - | 67.4 | - | 68.3 | - | 76.2 | - | 66.6 | -
MIMICK Zhao et al. (2018) | 9M | 517K | 69.7 | 62.3 | 73.6 | 61.4 | 68.0 | 65.2 | 56.6 | 56.7 | 64.2
BoS Zhao et al. (2018) | 500M | - | 79.7 | 72.6 | 73.6 | 69.5 | 79.5 | 68.6 | 66.4 | 61.5 | 71.5
KVQ-FH Sasaki et al. (2019) | 12M | - | 77.8 | 71.4 | 72.9 | 66.5 | 73.1 | 70.4 | 46.2 | 53.5 | 66.5
LOVE | 6.3M | 200K | 81.4 | 73.2 | 74.4 | 66.7 | 78.6 | 69.7 | 64.7 | 63.8 | 71.6
Table 3: Performance on the extrinsic tasks, measured as accuracy and F1 (five
runs of different learning rates) for text classification and NER,
respectively. Typos are generated by simulated errors of an OCR engine Ma
(2019). The speed of producing word vectors with Edit Distance and LOVE is
_380s/10K_ words and _0.9s/10K_ words, respectively.

##  5 Experiments

###  5.1 Evaluation Datasets

There are two main methods to evaluate word representations: Intrinsic and
Extrinsic. Intrinsic evaluations measure syntactic or semantic relationships
between words directly, e.g., word similarity in word clusters. Extrinsic
evaluations measure the performance of word embeddings as input features to a
downstream task, e.g., named entity recognition (NER) and text classification.
Several studies have shown that there is no consistent correlation between
intrinsic and extrinsic evaluation results Chiu et al. (2016); Faruqui et al.
(2016); Wang et al. (2019). Hence, we evaluate our representation by both
intrinsic and extrinsic metrics. Specifically, we use 8 intrinsic datasets (6
word similarity and 2 word cluster tasks): RareWord Luong et al. (2013),
SimLex Hill et al. (2015), MTurk Halawi et al. (2012), MEN Bruni et al.
(2014), WordSim Agirre et al. (2009), Simverb Agirre et al. (2009), AP
Almuhareb (2006) and BLESS Baroni and Lenci (2011). We use four extrinsic
datasets (2 text classification and 2 NER tasks): SST2 Socher et al. (2013),
MR Pang and Lee (2005), CoNLL-03 Sang and De Meulder (2003) and BC2GM Smith et
al. (2008). It is worth noting that the RareWord dataset contains many long-
tail words and the BC2GM is a domain-specific NER dataset. All data
augmentations and typo simulations are implemented by
NLPAUG111https://github.com/makcedward/nlpaug. Appendix B provides more
details on our datasets and experimental settings.

###  5.2 Results on Intrinsic Tasks

Table 2 shows the experimental results on 8 intrinsic tasks. Compared to other
mimick-like models, our model achieves the best average score across 8
datasets while using the least number of parameters. Specifically, our model
performs best on 5 word similarity tasks, and second-best on the word cluster
tasks. Although there is a gap between our model and the original FastText, we
find our performance acceptable, given that our model is 100x times smaller.

| SST2 | CoNLL-03 |
---|---|---|---
Typo Probability | original | 10% | 30% | 50% | 70% | 90% | original | 10% | 30% | 50% | 70% | 90% | Avg
Static Embeddings |
FastText | 82.3 | 68.2 | 59.8 | 56.7 | 57.8 | 60.3 | 86.4 | 81.6 | 78.9 | 73.9 | 70.2 | 63.4 | 70.0
FastText + LOVE | 82.1 | 79.8 | 74.9 | 74.2 | 68.8 | 67.2 | 86.3 | 84.7 | 81.8 | 77.5 | 73.1 | 71.3 | 76.8
Dynamical Embeddings |
BERT | 91.5 | 88.2 | 78.9 | 74.7 | 69.0 | 60.1 | 91.2 | 89.8 | 86.2 | 83.4 | 79.9 | 76.5 | 80.7
BERT + LOVE | 91.5 | 88.3 | 83.7 | 77.4 | 72.7 | 63.3 | 89.9 | 88.3 | 86.1 | 84.3 | 80.8 | 78.3 | 82.1
Table 4: Robust evaluation (five runs of different learning rates) on text
classification and NER under simulated post-OCR typos. We use uncased and
cased BERT-base model for SST2 and CoNLL-03, respectively.

###  5.3 Results on Extrinsic Tasks

Table 3 shows the results on four downstream datasets and their corrupted
version. In this experiment, we introduce another non-trivial baseline: Edit
Distance. For each corrupted word, we find the most similar word from a
vocabulary using edit distance and then use the pre-trained vectors of the
retrieved word. Considering the time cost, we only use the first 20K words
appearing in FastText (2M words) as reference vocabulary.

The typo words are generated by simulating post-OCR errors. For the original
datasets, our model obtains the best results across 2 datasets and the second-
best on NER datasets compared to other mimick-like models. For the corrupted
datasets, the performance of the FastText model decreases a lot and our model
is the second best but has very close scores with BoS consistently. Compared
to other mimick-like models, our model with 6.5M achieves the best average
score. Although Edit Distance can effectively restore the original meaning of
word, it is 400x times more time-consuming than our model.

Figure 5: Evaluation of different methods based on FastText under typos.

###  5.4 Robustness Evaluation

In this experiment, we evaluate the robustness of our model by gradually
adding simulated post-OCR typos Ma (2019). Table 4 shows the performances on
SST2 and CoNLL-03 datasets. We observe that our model can improve the
robustness of the original embeddings without degrading their performance.
Moreover, we find our model can make FastText more robust compared to other
commonly used methods against unseen words: a generic UNK token or character-
level representation of neural networks. Figure 5 shows the robustness check
of different strategies. FastText+LOVE has a consistent improvement on both
SST2 and CoNLL-03 datasets. At the same time, LOVE degrades the performance on
the original datasets only marginally if at all.

###  5.5 Ablation Study

We now vary the components in our architecture (input method, encoder and loss
function) to demonstrate the effectiveness of our architecture.

#### Input Method.

To validate the effect of our Mixed Input strategy, we compare it with two
other methods: using only the character sequence or only the subword sequence.
Table 5 shows that the Mixed method achieves better representations, and any
removal of char or subword information can decrease the performance.

| parameters | RareWord | SST2
---|---|---|---
| embedding | others |  |
The original LOVE | 6.3M | 200K | 42.2 | 81.4
Varying the input method
only use Char | 299K | 200K | 17.7 | 71.5
only use Subword | 6.0M | 200K | 25.3 | 76.0
Varying the encoder
replace PAM with CNN | 6.3M | 270K | 28.4 | 61.1
replace PAM with RNN | 6.3M | 517K | 27.2 | 67.2
replace PAM with SA | 6.3M | - | 36.9 | 78.7
Varying the loss function
use MSE | 6.3M | 200K | 34.5 | 76.0
use $\ell_{au}(\lambda=1.0)$ | 6.3M | 200K | 40.8 | 80.8
Ablation of data augmentation and hard negatives
w/out hard negatives | 6.3M | 200K | 37.7 | 78.6
w/out hard negatives |  |  |  |
and augmentation | 6.3M | 200K | 37.8 | 78.2
Table 5: Ablation studies for the architecture of LOVE, measured as Spearman’s
$\rho$ and accuracy, respectively.

#### Encoder.

To encode the input sequence, we developed the Positional Attention Module
(PAM), which first extracts ngram-like local features and then uses self-
attention combine them without distance restrictions. Table 5 shows that PAM
performs the best, which validates our strategy of incorporating both local
and global parts inside a word. At the same time, the number of parameters of
PAM is acceptable in comparison. We visualize the attention weights of PAM in
Appendix C.4, to show how the encoder extracts local and global morphological
features of a word.

#### Loss Function.

LOVE uses the contrastive loss, which increases alignment and uniformity. Wang
and Isola (2020) proves that optimizing directly these two metrics leads to
comparable or better performance than the original contrastive loss. Such a
loss function can be written as:

$\ell_{\text{au}}=\ell_{\text{align}}+\lambda\cdot\ell_{\text{uniform}}$ (8)

Here, $\lambda$ is a hyperparameter that controls the impact of
$\ell_{\text{uniform}}$. We set this value to 1.0 because it achieves the best
average score on RareWord and SST2. An alternative is to use the Mean Squared
Error (MSE), as in mimick-like models. Table 5 compares the performances of
these different loss functions. The contrastive loss significantly outperforms
the MSE, and there is no obvious improvement by directly using alignment and
uniformity. We also tried various temperatures $\tau$ for the contrastive
loss, and the results are shown in Table A3 in the appendix. In the end, a
value of $\tau=0.07$ provides a good performance.

#### Data Augmentation and Hard Negatives.

In Table 5, we observe that the removal of our hard negatives decreases the
performance, which demonstrates the importance of semantically different words
with similar surface forms.

| SST2
---|---
typos per sentence | typo-0 | typo-1 | typo-2 | typo-3
BERT | 91.5 | 77.2 | 73.2 | 69.4
Mimicking Input Embeddings |  |  |  |
BERT + Add | 91.3 | 77.2 | 73.5 | 70.7
BERT + Linear Fukuda et al. (2020) | 91.4 | 79.6 | 77.2 | 72.8
BERT + Replacement | 91.5 | 81.4 | 78.7 | 73.6
Mimicking Distilled Embeddings |  |  |  |
BERT + Add | 91.3 | 78.8 | 75.6 | 72.3
BERT + Linear Fukuda et al. (2020) | 91.3 | 81.4 | 78.7 | 73.6
BERT + Replacement | 91.4 | 81.5 | 78.9 | 73.8
Table 6: Performances of different strategies that work with BERT together,
measured as the accuracy among five different learning rates.

LOVE uses five types of word augmentation. We find that removing this
augmentation does not deteriorate performance too much on the word similarity
task, while it causes a 0.4 point drop in the text classification task (the
last row in Table 5), where data augmentations prove helpful in dealing with
misspellings. We further analyze the performance of single and composite
augmentations on RareWord and SST2 in the appendix in Figure A3 and Figure A4.
We find that a combination of all five types yields the best results.

###  5.6 The performance of mimicking BERT

As described in Section 4.5, we can mimic the input or distilled embeddings of
BERT. After learning from BERT, we use the vectors generated by LOVE to
replace the embeddings of OOV subwords. Finally, these new representations are
fed into the multi-layer attentions. We call this method the _Replacement_
strategy. To valid its effectiveness, we compare it with two other baselines:
(1) Linear Combination Fukuda et al. (2020). For each subword
$\mathbf{e}^{sub}$, the generated vectors of word $\mathbf{e}^{word}$
containing the subwords are added to the subword vectors of BERT:

$\displaystyle\mathbf{e}^{new}$
$\displaystyle=(1-\alpha)~{}\mathbf{e}^{sub}+\alpha~{}\mathbf{e}^{word}$ (9)
$\displaystyle\alpha$
$\displaystyle=\text{sigmoid}~{}(\mathbf{W}\cdot\mathbf{e}^{sub})$

where $\mathbf{e}^{sub}\in\mathbb{R}^{d}$ is a subword vector of BERT, and
$\mathbf{e}^{word}\in\mathbb{R}^{d}$ is a generated vector of our model.
$\mathbf{W}\in\mathbb{R}^{d}$ are trainable parameters.

(2) Add. A generated word vector is directly added to a corresponding subword
vector of BERT:

$\displaystyle\mathbf{e}^{new}$
$\displaystyle=\mathbf{e}^{sub}+\mathbf{e}^{word}$ (10)

Table 6 shows the result of these strategies. All of them can bring a certain
degree of robustness to BERT without decreasing the original capability, which
demonstrates the effectiveness of our framework. Second, the replacement
strategy consistently performs best. We conjecture that BERT cannot restore a
reasonable meaning for those rare and misspelling words that are tokenized
into subwords, and our generated vectors can be located nearby the original
word in the space. Third, we find mimicking distilled embeddings performs the
best while mimicking input embeddings comes close. Considering that the first
method needs training on large-scale data, mimicking the input embeddings is
our method of choice.

##  6 Conclusion

We have presented a lightweight contrastive-learning framework, LOVE, to learn
word representations that are robust even in the face of out-of-vocabulary
words. Through a series of empirical studies, we have shown that our model
(with only 6.5M parameters) can achieve similar or even better word embeddings
on both intrinsic and extrinsic evaluations compared to other mimick-like
models. Moreover, our model can be added to models with static embeddings
(such as FastText) or dynamical embeddings (such as BERT) in a plug-and-play
fashion, and bring significant improvements there. For future work, we aim to
extend our model to languages other than English.

##  7 Acknowledgements

We sincerely thank all the reviewers for their insightful comments and helpful
suggestions. This work was partially funded by ANR-20-CHIA-0012-01 (“NoRDF”).

## References

  * Agirre et al. (2009) Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009.  A study on similarity and relatedness using distributional and wordnet-based approaches.
  * Almuhareb (2006) Abdulrahman Almuhareb. 2006.  _Attributes in lexical acquisition_.  Ph.D. thesis, University of Essex.
  * Baroni and Lenci (2011) Marco Baroni and Alessandro Lenci. 2011.  How we blessed distributional semantic evaluation.  In _Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics_ , pages 1–10.
  * Becker and Hinton (1992) Suzanna Becker and Geoffrey E Hinton. 1992.  Self-organizing neural network that discovers surfaces in random-dot stereograms.  _Nature_ , 355(6356):161–163.
  * Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018.  Synthetic and natural noise both break neural machine translation.  In _International Conference on Learning Representations_.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting pretrained contextualized representations via reductions to static embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4758–4781.
  * Bromley et al. (1993) Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993.  Signature verification using a “siamese” time delay neural network.  _International Journal of Pattern Recognition and Artificial Intelligence_ , 7(04):669–688.
  * Bruni et al. (2014) Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014.  Multimodal distributional semantics.  _Journal of artificial intelligence research_ , 49:1–47.
  * Cao and Rei (2016) Kris Cao and Marek Rei. 2016.  A joint model for word embedding and word morphology.  In _Proceedings of the 1st Workshop on Representation Learning for NLP_ , pages 18–26.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Chen et al. (2017) Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. 2017.  On sampling strategies for neural network-based collaborative filtering.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 767–776.
  * Chiu et al. (2016) Billy Chiu, Anna Korhonen, and Sampo Pyysalo. 2016.  Intrinsic evaluation of word vectors fails to predict extrinsic performance.  In _Proceedings of the 1st workshop on evaluating vector-space representations for NLP_ , pages 1–6.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * El Boukkouri et al. (2020) Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020.  Characterbert: Reconciling elmo and bert for word-level open-vocabulary representations from characters.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6903–6915.
  * Faruqui et al. (2016) Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016.  Problems with evaluation of word embeddings using word similarity tasks.  _arXiv preprint arXiv:1605.02276_.
  * Fukuda et al. (2020) Nobukazu Fukuda, Naoki Yoshinaga, and Masaru Kitsuregawa. 2020.  Robust backed-off estimation of out-of-vocabulary embeddings.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 4827–4838.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_.
  * Giorgi et al. (2020) John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. 2020.  Declutr: Deep contrastive learning for unsupervised textual representations.  _arXiv preprint arXiv:2006.03659_.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. 2020.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_.
  * Gupta and Jaggi (2021) Prakhar Gupta and Martin Jaggi. 2021.  Obtaining better static word embeddings using contextual embedding models.  _arXiv preprint arXiv:2106.04302_.
  * Halawi et al. (2012) Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012.  Large-scale learning of word relatedness with constraints.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 1406–1414.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738.
  * Heinzerling and Strube (2017) Benjamin Heinzerling and Michael Strube. 2017.  Bpemb: Tokenization-free pre-trained subword embeddings in 275 languages.  _arXiv preprint arXiv:1710.02187_.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Hjelm et al. (2018) R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018.  Learning deep representations by mutual information estimation and maximization.  _arXiv preprint arXiv:1808.06670_.
  * Hu et al. (2019) Ziniu Hu, Ting Chen, Kai-Wei Chang, and Yizhou Sun. 2019.  Few-shot representation learning for out-of-vocabulary words.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4102–4112.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8018–8025.
  * Jinman et al. (2020) Zhao Jinman, Shawn Zhong, Xiaomin Zhang, and Yingyu Liang. 2020.  Pbos: Probabilistic bag-of-subwords for generalizing word embedding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 596–611.
  * Kim et al. (2018) Yeachan Kim, Kang-Min Kim, Ji-Min Lee, and SangKeun Lee. 2018.  Learning to generate word representations using subword information.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2551–2561.
  * Li et al. (2018) Bofang Li, Aleksandr Drozd, Tao Liu, and Xiaoyong Du. 2018.  Subword-level composition functions for learning word embeddings.  In _Proceedings of the second workshop on subword/character level models_ , pages 38–48.
  * Liang et al. (2018) Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018\.  Deep text classification can be fooled.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 4208–4215.
  * Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Ro{bert}a: A robustly optimized {bert} pretraining approach.
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.  In _International Conference on Learning Representations_.
  * Luong et al. (2013) Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013.  Better word representations with recursive neural networks for morphology.  In _Proceedings of the seventeenth conference on computational natural language learning_ , pages 104–113.
  * Ma (2019) Edward Ma. 2019.  Nlp augmentation.  https://github.com/makcedward/nlpaug.
  * Ma et al. (2020) Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020\.  Charbert: Character-aware pre-trained language model.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 39–50.
  * MacQueen et al. (1967) James MacQueen et al. 1967.  Some methods for classification and analysis of multivariate observations.  In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_ , volume 1, pages 281–297. Oakland, CA, USA.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  _arXiv preprint cs/0506075_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237.
  * Piktus et al. (2019) Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Édouard Grave, Rui Ferreira, and Fabrizio Silvestri. 2019.  Misspelling oblivious word embeddings.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3226–3234.
  * Pinter et al. (2017) Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017.  Mimicking word embeddings using subword rnns.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 102–112.
  * Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019.  Combating adversarial misspellings with robust word recognition.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5582–5591.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Sasaki et al. (2019) Shota Sasaki, Jun Suzuki, and Kentaro Inui. 2019.  Subword-based compact reconstruction of word embeddings.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3498–3508.
  * Schick and Schütze (2019a) Timo Schick and Hinrich Schütze. 2019a.  Attentive mimicking: Better word embeddings by attending to informative contexts.  _arXiv preprint arXiv:1904.01617_.
  * Schick and Schütze (2019b) Timo Schick and Hinrich Schütze. 2019b.  Learning semantic representations for novel words: Leveraging both form and context.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6965–6973.
  * Schick and Schütze (2020) Timo Schick and Hinrich Schütze. 2020.  Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8766–8774.
  * Smith et al. (2008) Larry Smith, Lorraine K Tanabe, Rie Johnson nee Ando, Cheng-Ju Kuo, I-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M Friedrich, Kuzman Ganchev, et al. 2008.  Overview of biocreative ii gene mention recognition.  _Genome biology_ , 9(2):1–19.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Sun et al. (2020) Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020.  Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert.  _arXiv preprint arXiv:2003.04985_.
  * Üstün et al. (2018) Ahmet Üstün, Murathan Kurfalı, and Burcu Can. 2018.  Characters or morphemes: How to represent words?  Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2019) Bin Wang, Angela Wang, Fenxiao Chen, Yuncheng Wang, and C-C Jay Kuo. 2019.  Evaluating word embedding models: Methods and experimental results.  _APSIPA transactions on signal and information processing_ , 8.
  * Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR.
  * Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016.  Charagram: Embedding words and sentences via character n-grams.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1504–1515.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020\.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Wu et al. (2020) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020.  Clear: Contrastive learning for sentence representation.  _arXiv preprint arXiv:2012.15466_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Zhang and Wallace (2015) Ye Zhang and Byron Wallace. 2015.  A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1510.03820_.
  * Zhang et al. (2019) Yijia Zhang, Qingyu Chen, Zhihao Yang, Hongfei Lin, and Zhiyong Lu. 2019.  Biowordvec, improving biomedical word embeddings with subword information and mesh.  _Scientific data_ , 6(1):1–9.
  * Zhao et al. (2018) Jinman Zhao, Sidharth Mudgal, and Yingyu Liang. 2018.  Generalizing word embeddings using bag of subwords.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 601–606.
  * Zhu et al. (2019) Yi Zhu, Ivan Vulić, and Anna Korhonen. 2019.  A systematic study of leveraging subword information for learning word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 912–932.

##  Appendix A Details of Our Approach

###  A.1 Shrinking Our Model

We consider the following four methods to reduce the total parameters of our
model:

(1) Matrix Decomposition. The original matrix can be decomposed into two
smaller matrices
$\mathbf{V}=\mathbf{U}\times\mathbf{M},\mathbf{U}\in\mathbb{R}^{|\mathcal{V}|\times
h},\mathbf{M}\in\mathbb{R}^{h\times m}$ and $h<m$. Here, we set $m=300$ and
$h=200$ respectively.

(2) Top Subword. We use only the top-k frequent subwords, using the word
frequencies from a corpus. We set the parameter $k=20000$.

(3) Hashing. We use a hashing strategy to share memory for subwords aiming to
reduce the parameters. We use a bucket size of $20000$.

(4) Preprocessing. The original vocabulary contains plurals and conjugations,
therefore we stem all complete words and remove words with numerals, obtaining
a new vocabulary of $21257$ words.

Table A1 shows that the preprocessing method can reduce parameters very
effectively while obtaining a very competitive performance.

| parameters | RareWord | SST2
---|---|---|---
| embedding | non-embedding |  |
Original | 9M | 200K | 43.5 | 80.7
Decomposition | 5.6M | 200K | 38.1 | 80.3
Top-K | 6M | 200K | 39.2 | 80.1
Hashing | 6M | 200K | 40.5 | 80.4
Preprocessing | 6.3M | 200K | 42.4 | 80.7
Table A1: Performance of different shrinkage strategies, measured as Spearman’s $\rho$ and accuracy, respectively. The target vectors are from fasttext-crawl-300d-2M. Hyperparam | SST2 | MR | CONLL-03 | BC2GM
---|---|---|---|---
model | CNN | CNN | BiLSTM+CRF | BiLSTM+CRF
layer | 1 | 1 | 1 | 1
kernel | [3,4,5] | [3,4,5] | - | -
filter | 100 | 100 | - | -
hidden size | 300 | 300 | 300 | 300
optimizer | Adam | Adam | SGD | SGD
dropout | 0.5 | 0.5 | 0.5 | 0.5
batch size | 50 | 50 | 768 | 768
epoch | 5 | 5 | 100 | 100
Table A2: Hyperparameters for extrinsic datasets. Figure A1: An illustration
of our Mixed input for the word misspell. Figure A2: PCA visualizations of
word vectors generated by LOVE, BoS, and KVQ-FH. Different colors mean
different clusters, as predicted by K-means. There are three OOV words: oxgen,
archiitect and leukamia.

##  Appendix B Details of Our Experiments

###  B.1 Training of Mimick-like Models

Our target pre-trained embeddings are those from FastText fasttext-
crawl-300d-2M, because they provide a strong baseline. They sum up subword-
level information to produce vectors for arbitrary words. We also compare to
MIMICK, BoS, and KVQ-FH, which do not train on contextual words. We do not
compare to Robust Backed-off Estimation Fukuda et al. (2020) and PBoS Jinman
et al. (2020), because they need larger and more complex models. Robust
Backed-off Estimation uses string matching to find the top-k similar words
from the entire vocabulary when imputing. Using the same target vectors, the
number of parameter of BoS and PBoS are 163M and 316M, respectively. We re-
train MIMICK, BoS, and KVQ-FH as baselines according to the published
settings. In order to compare at the same parameter level, we use subwords for
MIMICK instead of pure characters and adjust the hashing size $H=40K$ for KVQ-
FH.

###  B.2 Robustness Evaluations

Figure A3: Performances of different augmentations on RareWord, measured as
Spearman’s $\rho$. Diagonal entries correspond to individual augmentation and
off-diagonal entries correspond to composite augmentation.

As for our model, we first lower-case and tokenize each word by using
WordPiece Wu et al. (2016) with a vocabulary of $30K$ subwords and preprocess
them by stemming and removing subwords with numerals. This yields a vocabulary
of $21257$ words. Each subword is represented by corresponding vectors from
our model and we adopt a modified attention model to encode the subword
sequence. Specifically, the layer number of this encoder is just $1$ for
efficiency and the hidden dimension is $300$. In each block, the number of
attention heads is $1$ and we use fixed sinusoidal position embeddings Vaswani
et al. (2017) for positional information. To train the contrastive learning
framework, we use the open-source tool Ma (2019) to augment a word, and use
the probabilities $\\{0.07,0.07,0.07,0.07,0.36,0.36\\}$ for six augmentations:
swap, drop, insert, keyboard, synonym, no-operation. Hard negatives are
generated by edit distance. For each target word, we store the top-100 similar
words and insert $3$ of them into a mini-batch as hard negatives. The loss
function is a standard contrastive loss with temperature $\tau=0.07$. The
optimizer is Adam and the learning rate is $0.002$. The dropout rate is $0.2$
and we train the model for $20$ epochs in total.

###  B.3 Intrinsic and Extrinsic Evaluations

We choose the setting discussed in Section 4 to train our model for 20 epochs,
and evaluate each intrinsic task based on the vectors that the models produce.
As for the extrinsic tasks, we feed word vectors into each neural network and
fix them during training. We use CNNs for text classification Zhang and
Wallace (2015) and BiLSTM+CRF for NER Huang et al. (2015). We compare
different embeddings on both intrinsic and extrinsic datasets by using
generated vectors. For the word cluster tasks, the produced vectors are
clustered by K-Means and then measured by Purity. The hyper-parameters of the
extrinsic tasks are shown in Table A2. For each dataset, our model is trained
with five learning rates $\\{5e-3,3e-3,1e-3,8e-4,5e-4\\}$. We select the best
one on the development set to report its score on the test set.

To generate a corrupted dataset, we simulate post-OCR errors. We adopt the
augmentation tool developed by Ma (2019) to corrupt $70\%$ of the original
words. To check the robustness of BERT, we directly finetune a BERT-base model
using Huggingface Wolf et al. (2020). During finetuning, the batch size is
$16$ and we train $5$ epochs. We select the best model among five learning
rates $\\{9e-5,7e-5,5e-5,3e-5,1e-5\\}$ on the development set and report the
score of the model on the test set.

Figure A4: Performances of different augmentations on SST2, measured as
accuracy. Diagonal entries correspond to individual augmentation and off-
diagonal entries correspond to composite augmentation.

###  B.4 Datasets

#### Intrinsic Datasets.

We use six word similarity datasets: (1) RareWord Luong et al. (2013) (2)
SimLex Hill et al. (2015) (3) MTurk Halawi et al. (2012) (4) MEN Bruni et al.
(2014) (5) WordSim Agirre et al. (2009), and (6) Simverb Agirre et al. (2009).
The task is scored by Spearman’s $\rho$, which computes the correlation
between gold similarity and the similarity obtained from generated vectors.
For the word cluster task, we use (1) AP Almuhareb (2006) and (2) BLESS Baroni
and Lenci (2011). The generated word vectors are first clustered by K-means
MacQueen et al. (1967) and then scored by the cluster purity.

#### Extrinsic Datasets.

We use both sentence-level and token-level downstream datasets to evaluate the
quality of word representations. For the sentence level, we use SST2 Socher et
al. (2013) and MR Pang and Lee (2005), and the metric is accuracy. For the
token level, we use two NER datasets: general CoNLL-03 Sang and De Meulder
(2003) and biomedical BC2GM Smith et al. (2008). The metric is the entity-
level F1 score. As before, we select the best model among five different
learning rates $\\{5e-3,3e-3,1e-3,8e-4,5e-4\\}$ on the development set and
then report the model score on the test set.

Figure A5: Visualization of positional weights for the post-OCR word bec0me
(the correct one is become). Figure A6: Visualization of self-attention
weights for the post-OCR word bec0me.

##  Appendix C Additional Analyses

###  C.1 Qualitative Analysis

To better understand the clusterings produced by LOVE, we chose 15 words from
the AP dataset Almuhareb (2006), covering three topics (Chemical Substance,
Illness, and Occupation). We added 3 corrupted words, oxgen, archiitect and
leukamia. Figure A2 shows how LOVE, BoS, and KVQ-FH cluster these words (using
a PCA projection and K-means). All approaches space out the clusters to some
degree. In particular, BoS and KVQ-FH have trouble separating professions and
chemical substances. For the corrupted words, only LOVE is able to embed them
close enough to their original form, so that they appear in the correct
cluster.

###  C.2 Effect of Augmentation for Text Classification

Figure A4 shows the performance of five augmentation strategies on the text
classification task SST2. We observe that synonym is the most effective
methods. The first four methods have a weaker effect, but the keyboard
replacement can bring a certain degree of improvement. The results on RareWord
are similar (Figure A3).

###  C.3 Effect of $\tau$ in Contrastive Loss

As discussed in Chen et al. (2020), a proper temperature can yield better
representation in the visual area because $\tau$ is able to weigh the
negatives by their relative hardness. As shown in Table A3, we attempt
different values of temperature and find that there is no consistent $\tau$
that makes a model work well both on intrinsic and extrinsic datasets. Hence,
we choose the best performer on average, i.e., $\tau=0.07$.

| parameters | RareWord | SST2
---|---|---|---
| embedding | non-embedding |  |
$\ell_{cl}$ ($\tau$ = 0.03) | 9M | 200K | 35.0 | 81.6
$\ell_{cl}$ ($\tau$ = 0.07) | 9M | 200K | 39.8 | 81.3
$\ell_{cl}$ ($\tau$ = 0.12) | 9M | 200K | 39.9 | 81.1
$\ell_{cl}$ ($\tau$ = 0.20) | 9M | 200K | 37.6 | 81.5
$\ell_{cl}$ ($\tau$ = 0.50) | 9M | 200K | 38.3 | 80.6
Table A3: Performances of contrastive loss with various temperature $\tau$,
measured as Spearman’s $\rho$ and accuracy respectively.

###  C.4 Visualization of Encoder

As mentioned before, we combine two types of attention heads (self-attention
and positional attention) to encode a subword sequence. Here, we visualize the
attention weights on each side and show how they work. Figure A5 shows the
position-dependent weights. We use sinusoidal functions to generate positional
embeddings, and the weights are the dot product between these embeddings. We
observe the positional weights tend to the left and right subwords in addition
to themselves, which yields trigram representations.

Figure A6 shows the self-attention weights which are computed from the trigram
subwords of positional attention. Hence, each subword in this figure is a
trigram representation instead of a single subword representation. As we see,
self-attention can capture global features regardless of distance. We take the
first token [CLS] as an example, and this self-attention assigns high weights
for the token e and [SEP], which constructs a representation like this: [CLS]b
$+$ me[SUB] $+$ ##me[SEP]. This segment tells us this word starts with b and
ends with me.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

# VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal
Framework

Chris Kelly Luhui Hu$\star$ Bang Yang$\star$ Stanford University Seeking AI
Peking University Yu Tian Deshun Yang Cindy Yang Harvard University Seeking AI
University of Washington, Seattle Zaoshan Huang Zihao Li Jiayin Hu Seeking AI
Seeking AI University of California, Los Angeles Yuexian Zou Peking University
Authors contributed equally<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With the emergence of large language models (LLMs) and vision foundation
models, how to combine the intelligence and capacity of these open-sourced or
API-available models to achieve open-world visual perception remains an open
question. In this paper, we introduce VisionGPT to consolidate and automate
the integration of state-of-the-art foundation models, thereby facilitating
vision-language understanding and the development of vision-oriented AI.
VisionGPT builds upon a generalized multimodal framework that distinguishes
itself through three key features: (1) utilizing LLMs (e.g., LLaMA-2) as the
pivot to break down users’ requests into detailed action proposals to call
suitable foundation models; (2) integrating multi-source outputs from
foundation models automatically and generating comprehensive responses for
users; (3) adaptable to a wide range of applications such as text-conditioned
image understanding/generation/editing. This paper outlines the architecture
and capabilities of VisionGPT, demonstrating its potential to revolutionize
the field of computer vision through enhanced efficiency, versatility, and
generalization, and performance. Our code and models will be made publicly
available.

###### Keywords:

VisionGPT Open-world visual perception Vision-language understanding Large
language model Foundation model

##  1 Introduction

In the rapidly evolving era characterized by generative AI, large language
models (LLMs) and vision foundation models have emerged as front runners in
advancing the research in natural language processing (NLP), computer vision
(CV), and their intersection[20, 30, 18, 16]. Although foundation models like
GPT-4 have established a remarkable benchmark, developing an interactive agent
to achieve open-world visual perception remains an ever-changing frontier
brimming with application potential[11, 26, 34], e.g., security surveillance
and autonomous driving.

(a) Without processing

(b) With YOLO detection

(c) With YOLO detection first and then SAM segmentation

Figure 1: An example of how YOLO and SAM collaborate to achieve instance
segmentation. Given image (a) and interested objects “fork” and “person”, YOLO
is applied for detection, based on which SAM can solely segment regions of
interest to improve efficiency. Figure 2: Overview of VisionGPT. It leverages
LLMs (e.g., Llama 2[29]) as the pivot to understand user intentions and state-
of-the-art foundation models as executors to produce desired output. Building
upon such a generalized multimodal framework, VisionGPT can accommodate any
up-to-date foundation models and facilitate collaboration between models to
address complex and challenging tasks. Figure 3: An example of the workflow of
VisionGPT, which can be broken down into four key steps: 1) request
understanding via LLMs, 2) foundation model selection, 3) execution, and 4)
post processing and integration.

In response to the demand for visual perception in open-world scenarios,
researchers are actively exploring task-oriented vision foundation models,
e.g., segment anything model[15] (SAM) and detect everything model[31].
However, each foundation model has its distinct knowledge, making it hard to
solve complex tasks with a single model. Instead, combining the intelligence
and capacity of the off-the-shelf models turns out to be a more efficient and
environment-friendly solution. Let’s consider the challenging instance
segmentation task, where distinct objects of the same class need to be
differentiated from each other through different colored masks, and take
YOLO[13, 2] and SAM as examples. As depicted in Fig. 1, we can unite YOLO and
SAM to achieve instance segmentation efficiently. Specifically, instead of
indiscriminately segmenting all objects in Fig. 1(a), we can first apply YOLO
to detect objects at a glance. Then given the interested objects “fork” and
“person”, we can retain some of the detection results shown in Fig. 1(b),
based on which SAM can generate unique masks for the interested objects (Fig.
1(c)). While the above example showcases the feasibility of model
collaboration, the automation of “divide and rule” for various user requests
and task scenarios remains unresolved.

In this paper, we introduce VisionGPT to enhance open-world visual perception
from a systems-building view. VisionGPT operates on a multimodal framework
that combines the strengths of state-of-the-art (SOTA) LLMs and vision
foundation models, including Llama 2[29], SAM-series[15, 38, 39], YOLO-
series[13, 2], DINO[22, 7], Detectron2[36], DALL-E[24, 5], CLIP[23], etc. Fig.
2 illustrates the cores of VisionGPT: it utilizes LLM as pivot to understand
the custom requests of users and automate the whole workflow whereas leverages
different foundation models as executors to generate desired outputs. The
workflow of our VisionGPT can be decomposed into a few steps:

  1. 1.

Request Understanding via LLMs: In this step, the LLM interprets the user’s
request and breaks down the user input into detailed action proposals. The
concomitant visual data is also uploaded to the whole system.

  2. 2.

Foundation Model Selection: Appropriate foundation models are selected
depending on the action proposals.

  3. 3.

Execution: Foundation models act as computing engines and provide their expert
insights to the uploaded visual data.

  4. 4.

Post Processing and Integration: The multi-source outputs from foundation
models are first post-processed, then integrated, and finally returned to the
user.

To better understand the workflow of VisionGPT, we give an example in Fig. 3.
Specifically, consider two distinct images rather than a single image as
input. A user might make the following request: “Find dogs and lemons in the
images and then highlight them only”. VisionGPT will have a LLM interpret this
request and break it down into several reasonable action proposals as below:

  1. A.

Locate dogs

  2. B.

Segment dogs

  3. C.

Locate lemons

  4. D.

Segment lemons

  5. E.

Integrate all results (optional)

After the user request has been interpreted, the best-suited foundation model
will be selected. In this case, action proposal A calls for a simple object
detection of all dogs, so models like YOLO will be called upon. Action
proposal B asks for the dogs to be highlighted, which means that the
segmentation abilities of SAM will be needed. As mentioned earlier, SAM can
leverage the detection results of dogs to reduce inference latency. Action
proposals C and D will repeat as the above.

It is noteworthy that VisionGPT is not limited to the models listed in Fig. 2,
i.e., VisionGPT is flexible to accommodate up-to-date foundation models for
better task-solving skills, making it adaptable to a wide range of
applications such as text-conditioned image understanding/generation/editing.
Moreover, VisionGPT serves multiple purposes that align with the current and
future needs of the AI community. By providing a generalized multimodal
framework, this project will help accelerate the development of vision-
oriented AI and bridge the gap between the status quo of LLMs and the emergent
CV multimodal paradigm. Last but not least, this paper will explore the
capabilities, methodologies, and potential future applications of this
technology.

##  2 Related Works

The integration of state-of-the-art (SOTA) computer vision (CV) technologies
and large language models (LLMs) has become increasingly prevalent in the
field of artificial intelligence. VisionGPT uniquely contributes to this
domain by synergizing SOTA vision models, such as YOLO [12] and Meta SAM [15],
into a comprehensive multimodal framework.

In terms of bridging frameworks with LLMs, HuggingGPT [27] emerges as a
pertinent reference, connecting the extensive model repository HuggingFace [8]
to LLMs like ChatGPT. Although this approach aligns with VisionGPT in
facilitating interactions with LLMs, VisionGPT distinguishes itself by
prioritizing a seamless integration and automation of SOTA vision models,
thereby fostering a robust and vision-focused AI ecosystem.

Grounded SAM [11] extends the capabilities of Meta SAM through training with
language instructions, enhancing its object segmentation proficiency. This
innovation resonates with VisionGPT’s objective of harnessing and amplifying
the unique capabilities of existing models to advance vision-oriented AI.

MiniGPT-4 [40] represents another stride towards integrating visual and
linguistic modalities, aligning a visual encoder with the advanced LLM,
Vicuna. Although there are parallels in modality integration between MiniGPT-4
and VisionGPT, VisionGPT stands out by offering a holistic integration and
automation of various SOTA vision models, ensuring versatility and peak
performance across diverse applications.

The work on VoxPoser [33] serves as a related endeavor to VisionGPT in
integrating Large Language Models (LLMs) with robotic manipulation tasks.
VoxPoser uniquely synthesizes robot trajectories and constructs 3D value maps
based on natural language instructions, highlighting its strength in dealing
with a wide array of manipulation tasks. VisionGPT, on the other hand, extends
the application of LLMs beyond manipulation, aiming to create a generalized
and automated framework for various vision-oriented tasks. Although they share
common ground in leveraging LLMs for robotic methods, each framework carves
out its niche, contributing uniquely to the intersection of language models
and vision-based robotic tasks.

In "Physically Grounded Vision-Language Models for Robotic Manipulation" [10],
the authors address the limitations of current Vision-Language Models (VLMs)
in understanding physical concepts crucial for robotic manipulation. They
introduce a new dataset, PhysObjects, to enhance the VLM’s comprehension of
these concepts, resulting in improved robotic planning performance. While
VisionGPT focuses on creating a unified and automated framework for a variety
of vision-based tasks using Large Language Models, [10] emphasizes physically-
grounding VLMs to augment their utility in robotic manipulation tasks. Both
works underscore the significance of integrating language models with visual
perception for robotic applications, albeit with different focuses and
methodologies.

Foundation models and GPTs grow so fast. There are other related works,
including Visual ChatGPT, WorldGPT, Flamingo, VLMo, VIOLET, and more [35, 37,
3, 4, 9]. They mainly focused on ChatGPT integration or transformer-based
solutions. VisionGPT differentiates them with its generalized multimodal
framework. Its own framework can fine-tune the LLM [28] for its specific
streamlining purpose [17].

In conclusion, despite the presence of related works in CV and LLM
integration, VisionGPT still offers unique capabilities. It does so by
unifying and automating the capabilities of SOTA vision models, optimizing
multimodal interactions, and delivering a streamlined and efficient user
experience.

##  3 VisionGPT

###  3.1 Overview

VisionGPT operates as a cooperative agent designed to facilitate vision-
language understanding and vision-oriented AI, reinventing our prior work[14].
VisionGPT combines the strengths of large language models (LLMs) and vision
foundation models, adopting an in-context learning approach to generalize and
automate a variety of complex tasks based on natural language instructions.
When confronted with a user’s request, VisionGPT initiates an automated
deployment of the complete workflow. This orchestrates the collaboration and
utilization of expert models, such as the YOLO model and the SAM model, to
successfully accomplish the designated objectives set up by the user.

Establishing a connection between LLMs and vision foundation models, VisionGPT
opens up a realm of possibilities that could potentially lead to the ultimate
customization of user requests. This synergy among foundation models creates a
dynamic ecosystem where the linguistic and reasoning capabilities of LLMs can
be seamlessly integrated with the image processing and understanding
capability of vision foundation models. When a user submits a request,
VisionGPT first utilizes LLMs to parse the natural language input to extract a
number of details, context, and intent and simultaneously analyzes any
accompanying images or visual data linked to the request. Then, VisionGPT
arranges collaboration among foundation models, effectively translating the
user’s request into scene understanding and object analysis within images or
visual data. This connection paves the way for a highly adaptive and context-
aware system, capable of adapting its responses to the user’s preferences,
language nuances, and the specific content of the visual data provided. As the
connection between VisionGPT and future LLMs evolves, the potential for
ultimate customization of user requests becomes increasingly tangible, such as
performing specific tasks relating to object recognition, scene understanding,
or even creative image generation, tailored to the user’s unique needs.

VisionGPT has four main components: APIs, Streamline AI, Verify and Generate,
and Fine Tuning. Please refer to Fig. 2 for the first 3 components.

  1. 1.

APIs: VisionGPT supports two types of APIs: specific APIs and generalized
APIs. The specific APIs can be some simple or standard APIs for some common
vision AI operations, for example, $\mathtt{labelObjects}$($<\mathtt{object}$
$\mathtt{name}>$, $<\mathtt{image}$ $\mathtt{location}>$). The generalized
APIs are flexible for inputs: a text prompt for instruction and a list of
images or videos. The prompt can control and instruct the operations on images
or videos.

  2. 2.

Streamline AI: VisionGPT has intelligent logic to automate the process of
vision-language understanding based on LLMs and the integrated SOTA vision
foundation models.

  3. 3.

Verify and Generate: VisionGPT is unique to verify the results against the
inputs for the best results. It will retry if it detects something wrong. For
example, it can use vision-text similarity measured by CLIP[23] as a proxy to
decide whether to rerun. The generation step is an additional step to use
other vision tools, such as OpenCV and GAI (Generative AI) to generate based
on the API requirements or the input prompts.

  4. 4.

Fine Tune: Although the LLM used in our VisionGPT, i.e., Llama 2, is one of
the best open-source LLMs, it still has limits in the specific domains and
streamlining generalization. So VisionGPT has its dedicated database and
historical vision-related dataset to fine-tune the Llama 2 for a better
domain-specific LLM.

VisionGPT is an open framework from the APIs to the internal streamlining
logic. The goal of VisionGPT aims to provide a generalized multimodal
framework for streamlining versatile vision AI.

###  3.2 Problem Formulation

Consider a scenario where we are given a natural language instruction $L$ that
describes a specific task. The goal of VisionGPT is to interpret $L$ and
translate it into a set of action proposals $A=\\{a_{1},a_{2},\dots,a_{N}\\}$,
where each element represents an individual operation such as object
recognition, image segmentation, or feature extraction. The core challenge is
to ensure that the interpretation of $L$ accurately reflects the intended
operations, aligning the output with the user’s expectations. The problem can
be mathematically formulated as:

$\min_{\mathcal{L}}\mathcal{L}(f(L),A)+\mathcal{R}(A),$ (1)

where $f(\cdot)$ denotes the inference process of the LLM (i.e., Llama 2 in
this work), $f(L)$ the generated action proposals, $\mathcal{L}(\cdot)$ the
loss function measuring the discrepancy between the generated and ground-truth
action proposals, and $\mathcal{R}(A)$ the regularization term ensuring the
action proposals are well-defined and executable. Given that it could be
difficult to minimize $\mathcal{L}(f(L),A)$ without downstream fine-tuning, we
further introduce a piece of system information $S$ to steer the LLM to
generate desired outputs. Thus, Eq. 1 can be modified to

$\min_{\mathcal{L}}\mathcal{L}(f(\\{S;L\\}),A)+\mathcal{R}(A),$ (2)

where $\\{;\\}$ denotes the text concatenation operation.

###  3.3 Action Proposal Generation via In-Context Learning

As indicated by Eq. 2, VisionGPT utilizes an advanced LLM to generate a
corresponding set of action proposals ($A$) based on a natural language
instruction ($L$) and a piece of system information ($S$). Motivated by the
findings of previous works [6, 32, 19] that LLMs’ performance can be improved
by in-context learning, we implement $S$ as the demonstration of several
paired $(L,A)$ examples to help the LLM understand the task format of action
proposal generation. One example is given in Tab. 1. In-context learning
ensures that the generated action proposals are better grounded in the
linguistic context provided by the user, thus facilitating the accuracy of
action proposal generation.

Table 1: An example of the LLM input for action proposal generation. LLM Input
(System Information ($S$) & User Input ($L$))
---
Please generate action proposals based on the following operation candidates:
"locate", "segment", …. The output should be as concise as possible.
Input: highlight dogs and frogs in the image
Output: "locate" dogs; "segment" dogs; "locate" frogs, "segment" frogs;
…
Input: [User Input]
Output:

###  3.4 Task Automation via Few-Shot Generalization

VisionGPT is designed to generalize across a wide array of tasks without
relying on excessive task-specific training data. The recent work [21]
indicates that pre-trained language models can benefit from task-specific
instructions when evaluated in terms of generalization to unseen tasks. Such
cross-task generalization capacity ensures that LLM-based VisionGPT can
automate action proposal generation across various contexts and applications
based on only a few task-specific in-context examples, showcasing impressive
efficiency and adaptability.

###  3.5 Joint Optimization for Coherent Task Execution

To guarantee that the generated action proposals are not only linguistically
aligned but also lead to successful execution, VisionGPT employs a joint
optimization strategy. This approach considers both the semantic congruence
between $L$ and $A$ and the practical feasibility of the action proposals.
Through this comprehensive optimization, VisionGPT ensures coherent and
effective task execution, aligning responses with the user’s intent.

(a) Find the guitar and segment it

(b) Find the yellow flower and segment it

(c) Find an animal and mask it

(d) Mask any building in the image

(a) Highlight all frogs by masking them

(b) Find a different animal and segment it

(c) Identify any anomaly object

(d) Find any anomaly object and highlight it

Figure 7: Text-conditioned image understanding results. For each subfigure,
the caption and the left image are user inputs whereas the right image is the
final response of VisionGPT. As we can see, VisionGPT can follow user
requirements to detect and segment correct objects, even though in some cases,
i.e., (f), (g), and (h), users do not specify an object name.

##  4 Experiments

In this section, we showcase how VisionGPT address different tasks by
integrating state-of-the-art foundation models.

#### Text-Conditioned Image Understanding.

By integrating the intelligence and capacities of Llama 2[29], YOLO series[2,
13], and SAM[15], our VisionGPT can highlight the desired object in complex
realistic scenes (Figs. 5(a), 5(b) and 5(c)) and animation scenes (Fig. 5(d)),
achieve instance segmentation (Fig. 7(a)), and detect correct objects even
without given a specific object name (Figs. 7(b), 7(c) and 7(d)).

#### Text-Conditioned Image Generation/Editing.

As shown in Fig. 11, our VisionGPT can edit the given images based on user
requests through the additional integration of Stable Diffusion models[25, 1].

(a) Please cover the naked capacitors

(b) Find the QR Code and generate a new one

(c) Replace the front car with a different type

(d) Please have a icy highway

(a) Please generate a mountain far away

(b) A big mountain at the end of the road

(c) It is so cold outside

(d) It is going to rain soon

Figure 11: Text-conditioned image generation/editing results. For each
subfigure, the caption and the left image are user inputs whereas the right
image is the final response of VisionGPT. As we can see, VisionGPT can
generate image outputs aligned with user custom requests.

##  5 Limitations

While VisionGPT presents a substantial advancement in integrating state-of-
the-art (SOTA) computer vision models with large language models (LLMs), it is
not without its limitations. A primary constraint lies in the rapid evolution
of SOTA vision foundation models and vision-language models, which poses a
challenge to maintaining the relevance and effectiveness of VisionGPT. The
continual emergence of new models and techniques necessitates frequent updates
and adaptations to the VisionGPT framework to ensure compatibility and optimal
performance. Another limitation stems from VisionGPT’s reliance on the
integration of multiple expert models. While this design enhances versatility
and ensures the utilization of the best features from various models, it also
introduces complexity in model management and coordination. To ensure seamless
interaction and data flow between frequently evolving models, meticulous
design and maintenance are required. Additionally, while VisionGPT aims to act
as a vision-language model, its performance is contingent on the integrated
vision models’ and LLMs’ quality and capabilities. Any shortcomings or biases
in these underlying models could potentially propagate through the VisionGPT
framework, affecting the overall performance and reliability of the system.

In conclusion, although VisionGPT represents a significant stride toward a
unified and versatile AI system, it is not exempt from limitations. Addressing
the seamless integration and management of diverse expert models, alongside
navigating the challenges presented by rapid advancements in vision foundation
models, underscores crucial areas for future enhancement and development.

##  6 Conclusion

In this work, we have presented VisionGPT, a novel agent that integrates
state-of-the-art large language models (LLMs) with vision foundation models.
The integration of LLMs enables VisionGPT to extract context, details, and
intent from user inputs, translating them into precise action proposals,
whereas vision foundation models can collaborate within VisionGPT to achieve
better efficiency, versatility, and performance. Therefore, VisionGPT
automates the entire workflow from request understanding to response
generation, offering a robust and adaptable platform for vision-language
understanding and various vision-oriented AI tasks. Looking ahead, there lies
vast potential for VisionGPT to evolve and seamlessly integrate with
forthcoming LLMs and vision models, offering even more personalized and
context-aware interactions. This research establishes the foundation for
future advancements in this field, to consistently enhance and customize AI
systems to address the diverse and expanding requirements of users.

## References

  * [1] Stable diffusion public release, https://stability.ai/news/stable-diffusion-public-release
  * [2] Aharon, S., Louis-Dupont, Ofri Masad, Yurkova, K., Lotem Fridman, Lkdci, Khvedchenya, E., Rubin, R., Bagrov, N., Tymchenko, B., Keren, T., Zhilko, A., Eran-Deci: Super-gradients (2021)
  * [3] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., Simonyan, K.: Flamingo: a visual language model for few-shot learning (2022)
  * [4] Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O.K., Aggarwal, K., Som, S., Wei, F.: Vlmo: Unified vision-language pre-training with mixture-of-modality-experts (2022)
  * [5] Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., Jiao, Y., Ramesh, A.: Improving image generation with better captions. https://cdn.openai.com/papers/dall-e-3.pdf (2023)
  * [6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [7] Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers (2023)
  * [8] Face, H.: Hugging face: An open-source community for machine learning (2023), https://huggingface.co/
  * [9] Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: Violet : End-to-end video-language transformers with masked visual-token modeling (2022)
  * [10] Gao, J., Sarkar, B., Xia, F., Xiao, T., Wu, J., Ichter, B., Majumdar, A., Sadigh, D.: Physically grounded vision-language models for robotic manipulation (2023)
  * [11] Grounded-SAM Contributors: Grounded-Segment-Anything (Apr 2023), https://github.com/IDEA-Research/Grounded-Segment-Anything
  * [12] Jocher, G., Chaurasia, A., Qiu, J.: Yolov8 by ultralytics. https://github.com/ultralytics/ultralytics (2023)
  * [13] Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLO (Jan 2023), https://github.com/ultralytics/ultralytics
  * [14] Kelly, C., Hu, L., Yang, C., Tian, Y., Yang, D., Yang, B., Huang, Z., Li, Z., Zou, Y.: Unifiedvisiongpt: Streamlining vision-oriented ai through generalized multimodal framework (2023)
  * [15] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [16] Koh, J.Y., Fried, D., Salakhutdinov, R.R.: Generating images with multimodal language models. In: Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems. vol. 36, pp. 21487–21506. Curran Associates, Inc. (2023)
  * [17] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners (2023)
  * [18] Li, C.: Large multimodal models: Notes on cvpr 2023 tutorial. arXiv preprint arXiv:2306.14895 (2023)
  * [19] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55(9), 1–35 (2023)
  * [20] Min, B., Ross, H., Sulem, E., Veyseh, A.P.B., Nguyen, T.H., Sainz, O., Agirre, E., Heintz, I., Roth, D.: Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys 56(2), 1–40 (2023)
  * [21] Mishra, S., Khashabi, D., Baral, C., Hajishirzi, H.: Cross-task generalization via natural language crowdsourcing instructions. In: 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022. pp. 3470–3487. Association for Computational Linguistics (ACL) (2022)
  * [22] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual features without supervision (2023)
  * [23] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021)
  * [24] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation (2021)
  * [25] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [26] Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. In: Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems. vol. 36, pp. 38154–38180. Curran Associates, Inc. (2023)
  * [27] Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face (2023)
  * [28] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: Llama: Open and efficient foundation language models (2023)
  * [29] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C.C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P.S., Lachaux, M.A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T.: Llama 2: Open foundation and fine-tuned chat models (2023)
  * [30] Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al.: Internimage: Exploring large-scale vision foundation models with deformable convolutions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14408–14419 (2023)
  * [31] Wang, Z., Li, Y., Chen, X., Lim, S.N., Torralba, A., Zhao, H., Wang, S.: Detecting everything in the open world: Towards universal object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11433–11443 (2023)
  * [32] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
  * [33] Wenlong, H., Chen, W., Ruohan, Z., Yunzhu, L., Jiajun, W., Li, F.F.: Voxposer: Composable 3d value maps for robotic manipulation with language models (2023)
  * [34] Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N.: Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 (2023)
  * [35] Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N.: Visual chatgpt: Talking, drawing and editing with visual foundation models (2023)
  * [36] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [37] Yang, D., Hu, L., Tian, Y., Li, Z., Kelly, C., Yang, B., Yang, C., Zou, Y.: Worldgpt: A sora-inspired video ai agent as rich world models from text and image inputs (2024)
  * [38] Zhang, C., Han, D., Qiao, Y., Kim, J.U., Bae, S.H., Lee, S., Hong, C.S.: Faster segment anything: Towards lightweight sam for mobile applications (2023)
  * [39] Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., Wang, J.: Fast segment anything (2023)
  * [40] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models (2023)