# Adapting Transformer Language Models for Predictive Typing
in Brain-Computer Interfaces

Shijia Liu  David A. Smith
Khoury College of Computer Sciences
Northeastern University
Boston, MA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Brain-computer interfaces (BCI) are an important mode of alternative and
augmentative communication for many people. Unlike keyboards, many BCI systems
do not display even the 26 letters of English at one time, let alone all the
symbols in more complex systems. Using language models to make character-level
predictions, therefore, can greatly speed up BCI typing (Ghosh and
Kristensson, 2017). While most existing BCI systems employ character n-gram
models or no LM at all, this paper adapts several wordpiece-level Transformer
LMs to make character predictions and evaluates them on typing tasks. GPT-2
fares best on clean text, but different LMs react differently to noisy
histories. We further analyze the effect of character positions in a word and
context lengths.

##  1 Introduction

The use of state-of-the-art language models in facilitating typing tasks is
prominent in various computer applications nowadays. From autocomplete
features in email and text messaging apps to search suggestion functionalities
in many websites, language models have been playing an important roles in
reducing users’ typing time and thus enhancing their overall experience by
providing accurate typing predictions.

One distinctive use of language models in predictive typing is in brain-
computer interface (BCI) systems. BCI systems enable people who have lost
their motor functions and cannot communicate via common writing and speaking
channels to express themselves by brain waves (Birbaumer et al., 1999; Sellers
et al., 2010). Specifically, these systems observe signals, such as
electroencephalograms (EEG), and process them in order to detect users’
intended expressions.

During typing in BCI systems, candidate letters are often presented to users
in series, so it is desirable to show more likely target letters earlier to
save typing time. Thus, language models become useful since they are able to
make relatively accurate typing suggestions as well as correct errors (Ghosh
and Kristensson, 2017). Currently, studies on using language models to
facilitate BCI typing have limited scope: the language models examined are
mostly simple character n-gram models (Oken et al., 2014; Mora-Cortes et al.,
2014; Speier et al., 2016, 2017, 2018) or other finite-state approaches (Dudy
et al., 2018). Dong et al. (2019) shows an LSTM based language model for
typing prediction that is robust to noisy input, but it requires training on
noisy data.

In the past few years, large-scale pretrained language models have proved to
be best suited for many NLP tasks due to the significant performance
improvements they are able to achieve. Hence, these models are good candidates
to incorporate into BCI systems. In this scenario, we would like language
models to give accurate predictions on what users intend to type. For such a
task, autoregressive language models, which are trained to sequentially
generate text, have shown to be very effective. This class of models—starting
with GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019), and
Transformer-XL (Dai et al., 2019)—generate coherent paragraphs of text that
are increasingly hard to distinguish from human output.

For text generation tasks, these autoregressive language models are typically
evaluated on metrics such as perplexity (PPL), bits-per-character (BPC) or
word level accuracy. However, since in BCI systems candidate characters are
presented sequentially to users, the rankings of the target letters matter the
most. Thus, information retrieval measures such as mean reciprocal rank (MRR)
and recall at rank $k$ should be the main evaluation metrics when judging
predictive performance. It is essential to achieve high mean reciprocal rank
on the correct letters and high recall of the correct characters in the top
$k$ candidates presented to the users.

In this paper, therefore, we evaluate several current pretrained language
models on a predictive typing task. We chose models that are large enough to
have proven effective on several tasks in prior work, but small enough to run
on a single machine for deployment in a BCI system. The language models we
evaluate are all based on the transformer architecture (Vaswani et al., 2017),
which has become the standard choice for many NLP problems. We seek to predict
the next characters the users intend to type given the input so far. The
research questions we aim to answer are: (1) What is the overall performance
on character prediction of the different models that we evaluate? How does the
performance differ when predicting different character positions in words? (2)
How does the predictive performance vary with different context length (we
model context length as the number of complete words in given input)? (3) In
the presence of noise, how does the performance degradation diverge for
different models, and what are the reasons for such difference? Further, is
such degradation correlated with different context lengths?

Four different transformer language models are examined in order to address
these questions: Reformer (Kitaev et al., 2020), Transformer-XL (Dai et al.,
2019), GPT-2 (Radford et al., 2019) and GPT (Radford et al., 2018). We also
utilize a unigram baseline model to compare results. Two datasets we use to
run experiments on are the ALS Phraseset (Costello, 2014) and the Switchboard
Corpus (Godfrey et al., 1992).

Evaluation results indicate that GPT-2 gives best performance on both
evaluation datasets across all metrics with clean input text. Predicting the
first characters in words proves to be the most difficult for all transformer
models, where it generally gets easier when predicting the later parts of
words. Longer context length usually helps improve the predicting performance,
although not always for predicting the first characters in words. Finally,
different models react differently to input noise, where some of the best
performing models on clean text degrade significantly with the presence of
noise. With some variation, the performance degradation with noise is
generally correlated with context lengths.

##  2 Related Work

There have been multiple attempts to integrate language models into BCI
systems. Oken et al. (2014) utilizes a six-gram character level language
model, where the difference between the predicted probabilities of the target
letters and the top ranked letters is used to mark the levels of spelling
difficulty. Speier et al. (2016, 2017, 2018) is a series of work that
investigates how typing rate and accuracy could be improved with the help of
language models in BCI systems by carrying out actual user studies. Dudy et
al. (2018) proposes an FST based joint word-character language model that
incorporates ambiguous history into decoding in order to build a more robust
prediction system, which achieves promising results on perplexity and
predictive accuracy. Lastly, Dong et al. (2019) examines the usefulness of
training recurrent language models on noisy text since it makes the models
more robust to typical noisy input in BCI systems.

##  3 Models and Methods

###  3.1 Models

As stated above, we mainly consider pretrained transformer-based
autoregressive language models for character prediction. The models we use for
our experiments are Reformer, Transformer-XL, GPT-2 and GPT. We briefly review
the main ideas of these models below:

  * •

Reformer (Kitaev et al., 2020): Reformer is an autoregressive transformer
language model with techniques that allow improved computing efficiency. The
techniques include replacing dot-product attention by LSH (locality-sensitive
hashing) attention, as well as using reversible residual layers instead of
standard residuals. The model we use for our experiments is a character level
Reformer model trained on enwik8 (Hutter, 2018).

  * •

Transformer-XL (Dai et al., 2019): Transformer-XL introduces a novel
recurrence mechanism that allows the model to learn longer-term dependencies
beyond a fixed length context. The segment-level recurrence mechanism (a
segment is a number of consecutive tokens), which considers inputs from two
consecutive segments, enables the model to capture information from the
previous segment as well as the current one.

  * •

GPT-2 (Radford et al., 2019): GPT-2 is a 1.5B parameter transformer pretrained
on a large and diverse English corpus. It is trained to predict the next words
in sentences. The training process is autoregressive, meaning that the target
sequences of the model are the same as the input sequences, shifted one token
to the right. When making predictions, the model can only see input from
history, not from future. Hence, the model is suitable for text generation
tasks.

  * •

GPT (Radford et al., 2018): GPT is the previous version of GPT-2. It has the
same autoregressive training objective and is trained on the BookCorpus
dataset (Zhu et al., 2015). The model has 117M parameters.

###  3.2 Methods

In this section we detail the methods we use to generate character level
predictions from what is returned by language models in section 3.1.

####  3.2.1 Reformer: Direct Character Level Predictions

The Reformer model is a character level model, which returns a probabilistic
distribution over a set of characters in its vocabulary for next character
prediction. We simply use this distribution as final result.

####  3.2.2 Transformer-XL: Whole Word Prediction with a Closed Vocabulary

The Transformer-XL model utilizes a closed vocabulary of English words. We
simply take the input and remove the partially-typed last word. The model
gives a prediction of the last word over the entire vocabulary. Then we select
words in the vocabulary that have prefixes matching the partially-typed last
word from the input. Then we renormalize the probability distribution over the
selected words and marginalize over the first character after the matched
prefix. The resulting character-level distribution is a prediction of the next
character given current input.

####  3.2.3 GPT-2 and GPT: Beam Search on Subword Units

As for GPT-2 and GPT models, byte-pair encoding (BPE) (Sennrich et al., 2016)
is used for tokenization. The main idea of BPE is to create a fixed-size
vocabulary that contains common English subword units. A less common word
would thus be broken down into several subword units in the vocabulary. For
example, in GPT-2, the tokenization of character sequence _“peanut butter and
jel"_ would be (“_” denotes a space character):

_[’pe’, ’anut’, ’_butter’, ’_and’, ’_j’, ’el’]_

Therefore, in order to generate a predictive distribution on the next
characters, we need to examine all the possibilities that could complete the
final subword units in the input sequences. Similar to what we do for the
Transformer-XL model, we make the model predict the last subword unit over the
entire subword vocabulary. Then we select the subword units with prefixes
matching the partially-typed last subword from the input (in the example case
above, the selected subword units should start with _“el"_). We then perform
the same renormalization and marginalization as for the Transformer-XL model
and obtain a distribution over a set of characters.

However, predicting only the final subword units may limit the space of
possible continuations of the input. To remedy that, we need to look beyond
one subword unit. We therefore use beam search to generate multiple subword
units. Beam search is a heuristic search algorithm that expands the most
promising hypotheses in a selected set. In our case, the most promising
hypotheses would be the top ranked subword units on the current beam. For each
of these hypotheses, we make the language model further predict the next
subword unit over the entire vocabulary. Then the top ranked hypotheses
returned are again retained for further expansion. The renormalization and
marginalization are performed at the end of the beam search for all candidates
on the beam. The beam size and the beam search depth are hyperparameters that
can be adjusted.

##  4 Data and Experiments

###  4.1 Data

In our experiments we evaluate on two datasets. The first is closely related
to the BCI domain: it contains manually anonymized messages created by people
with ALS (Amyotrophic Lateral Sclerosis, a progressive nervous system disease
that causes loss of muscle control). The second dataset is the Switchboard
corpus (Godfrey et al., 1992), which is a collection of transcripts of two-
sided telephone conversations. We choose this dataset since we would like to
simulate the conversational nature of communications between BCI users and
systems. Below we will give more detailed information of these two datasets.

  * •

ALS Phraseset (Costello, 2014): The dataset contains 2,082 messages that serve
as a “vocabulary” in a message bank. The creation of these message is a part
of a process called _voice banking_. Voice banking is a procedure of
“recording a large inventory of someone’s speech which is then used to create
a synthetic voice that approximates their natural voice” (Costello, 2014).
This enables people to communicate messages via a voice synthesizer that
imitates their natural speech. The message vocabulary in this dataset can be
retrieved by users of augmentative communication technologies and they can
speak them in their own voice.

  * •

Switchboard Corpus (Godfrey et al., 1992): The Switchboard Corpus contains
transcripts from about 260 hours of speech. Specifically, it includes about
2,400 two-sided telephone conversations among 543 speakers from around the
United States. The corpus was collected automatically over T1 lines at Texas
Instruments. The callers are given appropriate recorded prompts, and topics
for discussion are also introduced before the conversations start.

###  4.2 Experiments

In this section, we describe the experiments performed for our analysis. We
first outline data preprocessing steps and introduce evaluation metrics. Then
we inspect results from various experiments in detail.

####  4.2.1 Data Preprocessing

For the ALS phraseset, we remove all the punctuation marks and expand all
contractions. As for the Switchboard corpus, we delete interjections such as
“uh-huh", “yeah" and “uh", as well as punctuation and dialogue-specific
placeholders such as “MUMBLEx”.

For evaluation, we use the whole ALS phraseset and randomly sample 500
conversations from Switchboard. For our experiments on character predictions,
we only look at the first four turns of the Switchboard conversations.

####  4.2.2 Evaluation Metrics

The main evaluation metrics for our character prediction task is MRR and
Recall@_k_. We report and compare MRR@_k_ and Recall@_k_ across different
models, where _k_ indicates the number of top-ranked characters in our
predictions that will be taken into account when computing the metrics.
Specifically, MRR@_k_ is the mean over different predictions of the reciprocal
rank of the correct target letter if it is in the top _k_ characters or zero
otherwise. Recall@_k_ indicates the proportion of the cases where the correct
target letter is in the top _k_ characters suggested by the language model.

###  4.3 Main Results on Character Prediction

In this section, we report the main results of our character prediction task
on the ALS Phraseset and the Switchboard Corpus. We examine results averaged
over all character positions in words, as well as results for different
character positions. For the ALS Phraseset, we let the language models predict
the characters in the last word of each phrase. For the Switchboard Corpus,
the last word of each turn is used for prediction. Furthermore, we use a
unigram language model trained on the ALS Phraseset to interpolate with all
the transformer models (except for the character level model Reformer) in our
experiments for smoothing purposes, since the distributions given by the
transformer models can be very “sharp", where most of the probability mass
falls on the top few letters. Therefore, interpolating with a unigram model
gives more probability to the lower ranked letters. We choose an interpolation
coefficient of 0.8. As for the beam search method we use for GPT-2 and GPT,
the beam size is chosen to be 20 where the search depth is 2. We use the
Huggingface transformers v4.17.0
111https://huggingface.co/docs/transformers/index implementations for all
transformer language models and run our experiments with their default
parameter settings. For all experiments, we report the mean results of 3
consecutive runs. We run our experiments on a computing cluster with GPUs and
it takes $\sim$9 hours to complete one run on both datasets.

Language Model | MRR@10 | Recall@10 | MRR@5 | Recall@5 | MRR@3 | Recall@3
---|---|---|---|---|---|---
Unigram Baseline | 0.2294 | 0.7022 | 0.1835 | 0.3681 | 0.1552 | 0.2424
Reformer | 0.7080 | 0.9381 | 0.6984 | 0.8664 | 0.6806 | 0.7883
Transformer-XL | 0.7267 | 0.9308 | 0.7185 | 0.8700 | 0.7039 | 0.8060
GPT: Beam Search | 0.7776 | 0.9487 | 0.7706 | 0.8973 | 0.7607 | 0.8545
GPT-2: Beam Search | 0.8253 | 0.9611 | 0.8196 | 0.9195 | 0.8103 | 0.8789
Table 1: Results on the ALS Phraseset, averaged across all character positions Language Model | MRR@10 | Recall@10 | MRR@5 | Recall@5 | MRR@3 | Recall@3
---|---|---|---|---|---|---
Unigram Baseline | 0.2292 | 0.7184 | 0.1821 | 0.3669 | 0.1543 | 0.2420
Reformer | 0.7361 | 0.9394 | 0.7265 | 0.8708 | 0.7115 | 0.8050
Transformer-XL | 0.7648 | 0.9419 | 0.7558 | 0.8756 | 0.7447 | 0.8270
GPT: Beam Search | 0.6474 | 0.8960 | 0.6316 | 0.7774 | 0.6180 | 0.7175
GPT-2: Beam Search | 0.8185 | 0.9530 | 0.8122 | 0.9055 | 0.8028 | 0.8650
Table 2: Results on the Switchboard Corpus, averaged across all character
positions

##### Averaging over all character positions in a word

Table 1 shows the results on the ALS phraseset, averaged over all character
positions. To give a straightforward sense of the good performance all
transformer language models deliver, we include a unigram baseline model
(trained on the ALS Phraseset). As we can see, all transformer language models
perform significantly better than the unigram baseline model. Among all
models, GPT-2 fares best across all metrics, achieving a 82.53% MRR@10 and a
96.11% Recall@10. Interestingly, the Reformer model, which is trained at the
character level, does not perform as well as other transformer models trained
at word (or subword) level. This is probably due to the fact that the corpus
that the Reformer model is trained on (enwik8) is relatively smaller and less
diverse compared to the training corpus of other transformer models.

Table 2 shows the results on the Switchboard Corpus, averaged over all
character positions. Same as what we see on the ALS phraseset, GPT-2 gives the
best performance across all metrics, achieving a 81.85% MRR@10 and a 95.30%
Recall@10. Interesting, GPT does not perform as well as it does on the ALS
phraseset. In fact, it has the worst results among all models. This is
probably because the BookCorpus dataset it is trained on (Zhu et al., 2015)
has little in common with conversational transcripts. The Reformer and the
Transformer-XL models perform reasonably well. Similar to the what we have for
the ALS phraseset, Reformer as a character based transformer model does not
fare as well as word based models such as Transformer-XL and GPT-2.

##### On specific positions in a word

Figure 1 and Figure 2 illustrate the results on MRR@10 and Recall@10 for
different character positions in the predicted words on the ALS Phraseset. As
shown in these figures, predicting the first character in a word proves to be
the most difficult across all transformer models. GPT-2 and GPT fare better in
predicting the first character than Reformer and Transformer-XL do. As we have
more characters appearing in a word as context, the performance starts to
improve for all transformer models, and the gap between models begins to
decrease. All transformer models achieves near-perfect results in Recall@10
when we have more than 8 characters typed in a word.

Figure 3 and Figure 4 show results for different character positions in the
predicted words on the Switchboard Corpus. Similar to what we have on the ALS
phraseset, predicting the first character in a word is the most difficult for
transformer models. When predicting earlier positions in words, there are
significant gaps between the results of GPT and that of other transformer
models. Such gaps contribute to the fact that GPT performs worst among all
transformer models on average over all character positions. The gaps
eventually diminish as we have longer context in predicted words. Although
both MRR@10 and Recall@10 approach 1 for transformer models towards the end of
predicted words, we do see a drop in MRR@10 from GPT and Transformer-XL when
the number of context character is 12, which is interesting.

Figure 1: MRR@10 for different character positions on ALS Phraseset Figure 2:
Recall@10 for different character positions on ALS Phraseset Figure 3: MRR@10
for different character positions on Switchboard Corpus Figure 4: Recall@10
for different character positions on Switchboard Corpus

###  4.4 The Role of Context Length

In this section we discuss how performance varies with different context
lengths in input history. We represent context length as the number of words
in the input prior to the words used for prediction. Since the context lengths
in the Switchboard corpus are scattered sparsely over a large range of
numbers, the results do not provide very meaningful interpretations.
Therefore, we only show results on the ALS Phraseset in this section.

Figure 5: MRR@10 for different number of context words on ALS Phraseset Figure
6: Recall@10 for different number of context words on ALS Phraseset

##### Average performance over all character positions

Figure 5 and Figure 6 show MRR@10 and Recall@10 averaged over all character
positions in predicted words for different number of context words. As we can
see, results on both metrics increase as we have more words in context for
transformer models. Of all transformer models, GPT-2 gives best results on
both MRR@10 and Recall@10 for almost all different context lengths, which is
aligned with our observations from section 4.3 that GPT2 fares best across all
models.

##### GPT-2’s performance vs. context length on predicting different character
positions of words

Figure 7 and Figure 8 illustrate MRR@10 and Recall@10 on predicting the first,
second, third and fourth characters in words versus different number of
context words from the GPT-2 model. As shown in the results, the performance
on predicting the first character is consistently the worst. The results on
both metrics also show a general improvement in performance when the character
position increases. There is no strict upward trend in MRR as we have more
context, especially for predicting the first character.

Figure 7: GPT2 model’s MRR@10 vs. context length on predicting different
character positions of words on ALS Phraseset Figure 8: GPT2 model’s Recall@10
vs. context length on predicting different character positions of words on ALS
Phraseset

###  4.5 Noisy Histories

In this section, we examine the effect of noise in typing history and how it
changes the prediction performance of different models. Noisy input is very
common in BCI settings since BCI users often make spelling mistakes. In our
experiment, we artificially replace a certain amount of context with randomly
selected letters from the English alphabet.

Table 3 and Table 4 show the decreases in performance metrics for various
models on the ALS Phraseset and on the Switchboard Corpus. Absolute numbers of
the performance metrics are in appendix A. On both datasets, Transformer-XL
exhibits the smallest degradation in performance among all models.
Furthermore, Transformer-XL shows the best absolute performance across all
metrics. This is probably due to the fact that, since Transformer-XL employs a
closed vocabulary, we do not use beam search to predict multiple subword
units. Instead we let the model predict the last word of input and compute
exact character marginals on words in the vocabulary that have prefixes
matching the partially typed last word. Therefore, the effect of noise is
smaller since the model considers longer input units (words instead of
subwords or individual characters). When a single incorrect character appears
in a relatively long whole word, it may not confuse the model very much. On
the other hand, both GPT and GPT-2 see significant drops in performance with
the presence of noise, indicating that these models are not very robust to
noisy input.

Figure 9 and Figure 10 illustrate the changes in MRR and Recall for different
models with respect to different context lengths (As in Section 4.4 above, we
only experiment on the ALS phraseset). In general, the drops in two metrics
increases when the context is longer. This is probably because more noise (in
an absolute sense) is introduced with longer input, which brings more negative
impact on predictions given by the language models. However, we do see a few
outliers where the changes drop sharply with longer context. This is an
interesting phenomenon, and we would like to explore more on it for future
work.

Language Model |  $\Delta$MRR@10 |  $\Delta$Recall@10 |  $\Delta$MRR@5 |  $\Delta$Recall@5 |  $\Delta$MRR@3 |  $\Delta$Recall@3
---|---|---|---|---|---|---
Reformer | -0.0903 | -0.0338 | -0.0947 | -0.0662 | -0.0990 | -0.0846
Transformer XL | -0.0401 | -0.0079 | -0.0423 | -0.0231 | -0.0448 | -0.0332
GPT: Beam Search | -0.2089 | -0.0777 | -0.2192 | -0.1546 | -0.2294 | -0.1999
GPT2: Beam Search | -0.1863 | -0.0631 | -0.1940 | -0.1201 | -0.2040 | -0.1633
Table 3: Results Difference on the ALS Phraseset with 10% noise present, averaged across all character positions Language Model |  $\Delta$MRR@10 |  $\Delta$Recall@10 |  $\Delta$MRR@5 |  $\Delta$Recall@5 |  $\Delta$MRR@3 |  $\Delta$Recall@3
---|---|---|---|---|---|---
Reformer | -0.1162 | -0.0453 | -0.1218 | -0.0891 | -0.1267 | -0.1097
Transformer XL | -0.0693 | -0.0245 | -0.0702 | -0.0319 | -0.0753 | -0.0532
GPT: Beam Search | -0.1607 | -0.0713 | -0.1677 | -0.1244 | -0.1756 | -0.1588
GPT2: Beam Search | -0.1935 | -0.0708 | -0.2013 | -0.1288 | -0.2093 | -0.1643
Table 4: Results Difference on the Switchboard Corpus with 10% noise present,
averaged across all character positions Figure 9: $\Delta$MRR@10 for different
number of context words on ALS Phraseset with 10% noise present Figure 10:
$\Delta$Recall@10 for different number of context words on ALS Phraseset with
10% noise present

##  5 Conclusion

In this paper we explore the performance of four transformer language models
(Reformer, Transformer-XL, GPT-2 and GPT) on predictive typing tasks, using
metrics relevant to their deployment in brain-computer interface systems. We
experiment on two datasets: the ALS Phraseset and the Switchboard corpus. We
also develop a beam search based algorithm for GPT-2 and GPT to predict
multiple subword units. We examine the experimental results averaged over all
character positions in a word, as well as on different character positions.
Among all the models, GPT-2 fares best in most scenarios. We also see that the
performance improves for all models on later character positions in a word.
Furthermore, we investigate the effect of context lengths in prediction
performance. Generally, the predictions are more accurate with longer context.
Lastly, we examine the effect of noise in input. It turns out that
Transformer-XL is the most noise-robust language model at the level of 10%
noise. On the other hand, the performance of GPT-2 and GPT are heavily
impacted by the presence of noise. The performance degradation with noise is
generally correlated with context lengths.

Future work will involve investigating the relationship between prediction
performance and the internal properties of language models. For example, we
explore how perplexity relates to these predictive evaluation metrics.
Moreover, we can see whether fine-tuning the language models with noisy input
could reduce the performance degradation with noisy context. Lastly, we are
happy to share that our collaborators have begun testing these language models
on real BCI systems. It would be interesting to see how actual users react to
the enhancements these language models provide. Overall, we believe that our
experiments open exciting avenues for this line of research.

##  6 Limitations

We have conducted an extensive evaluation of the performance of large language
models using the transformer architecture on character-level predictive typing
tasks. We note, however, a few areas of limitation that suggest avenues for
future work:

  * •

As noted above, models using subword tokenization seem particularly
susceptible to character typing errors in the history. More work needs to be
done exploring how character-level edits change tokenization, how they
interact with search errors in beam search, and how these cascading errors can
be mitigated.

  * •

Due to a lack of extensive, anonymized BCI typing logs, we use a character
unigram edit model to simulate noisy data. As more real BCI interaction data
becomes available, we could simulate more realistic noisy histories.

  * •

Analyzing typing accuracy in longer documents or long-running dialogues, where
transformer architectures can take advantage of more context, is left for
future work.

## References

  * Birbaumer et al. (1999) Niels Birbaumer, Nimr Ghanayim, Thilo Hinterberger, Iver Iversen, Boris Kotchoubey, Andrea Kübler, Juri Perelmouter, Edward Taub, and Herta Flor. 1999\.  A spelling device for the paralysed.  _Nature_ , 398(6725):297–298.
  * Costello (2014) John M Costello. 2014.  Message banking, voice banking and legacy messages.  _Boston, MA: Boston Children’s Hospital_.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Dong et al. (2019) Rui Dong, David A Smith, Shiran Dudy, and Steven Bedrick. 2019.  Noisy neural language modeling for typing prediction in bci communication.  In _Proceedings of the Eighth Workshop on Speech and Language Processing for Assistive Technologies_ , pages 44–51.
  * Dudy et al. (2018) Shiran Dudy, Steven Bedrick, Shaobin Xu, and David A Smith. 2018.  A multi-context character prediction model for a brain-computer interface.  In _Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting_ , volume 2018, page 72. NIH Public Access.
  * Ghosh and Kristensson (2017) Shaona Ghosh and Per Ola Kristensson. 2017.  Neural networks for text correction and completion in keyboard decoding.  _arXiv preprint arXiv:1709.06429_.
  * Godfrey et al. (1992) John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992.  Switchboard: Telephone speech corpus for research and development.  In _Acoustics, Speech, and Signal Processing, IEEE International Conference on_ , volume 1, pages 517–520. IEEE Computer Society.
  * Hutter (2018) Marcus Hutter. 2018.  The human knowledge compression contest.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_.
  * Mora-Cortes et al. (2014) Anderson Mora-Cortes, Nikolay V Manyakov, Nikolay Chumerin, and Marc M Van Hulle. 2014.  Language model applications to spelling with brain-computer interfaces.  _Sensors_ , 14(4):5967–5993.
  * Oken et al. (2014) Barry S Oken, Umut Orhan, Brian Roark, Deniz Erdogmus, Andrew Fowler, Aimee Mooney, Betts Peters, Meghan Miller, and Melanie B Fried-Oken. 2014.  Brain–computer interface with language model–electroencephalography fusion for locked-in syndrome.  _Neurorehabilitation and neural repair_ , 28(4):387–394.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sellers et al. (2010) Eric W Sellers, Theresa M Vaughan, and Jonathan R Wolpaw. 2010.  A brain-computer interface for long-term independent home use.  _Amyotrophic lateral sclerosis_ , 11(5):449–455.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Speier et al. (2016) William Speier, C Arnold, and Nader Pouratian. 2016.  Integrating language models into classifiers for bci communication: a review.  _Journal of neural engineering_ , 13(3):031002.
  * Speier et al. (2018) William Speier, Corey Arnold, Nand Chandravadia, Dustin Roberts, Shrita Pendekanti, and Nader Pouratian. 2018.  Improving p300 spelling rate using language models and predictive spelling.  _Brain-Computer Interfaces_ , 5(1):13–22.
  * Speier et al. (2017) William Speier, Nand Chandravadia, Dustin Roberts, Shrita Pendekanti, and Nader Pouratian. 2017.  Online bci typing using language model classifiers by als patients in their homes.  _Brain-Computer Interfaces_ , 4(1-2):114–121.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE international conference on computer vision_ , pages 19–27.

##  Appendix A Further Noisy Input Results

Table 5 and Table 6 show the absolute values of performance metrics with 10%
noise present. As we can see, Transformer XL delivers the best results among
all models on both datasets. Further, all transformer language models perform
significantly better than the unigram baseline even with the presence of
noise.

Language Model | MRR@10 | Recall@10 | MRR@5 | Recall@5 | MRR@3 | Recall@3
---|---|---|---|---|---|---
Unigram Baseline | 0.2294 | 0.7022 | 0.1835 | 0.3681 | 0.1552 | 0.2424
Reformer | 0.6177 | 0.9043 | 0.6037 | 0.8002 | 0.5816 | 0.7037
Transformer XL | 0.6866 | 0.9229 | 0.6762 | 0.8469 | 0.6591 | 0.7728
GPT: Beam Search | 0.5687 | 0.8710 | 0.5514 | 0.7427 | 0.5313 | 0.6546
GPT2: Beam Search | 0.6390 | 0.8980 | 0.6256 | 0.7994 | 0.6063 | 0.7156
Table 5: Results on the ALS Phraseset with 10% noise present, averaged across all character positions Language Model | MRR@10 | Recall@10 | MRR@5 | Recall@5 | MRR@3 | Recall@3
---|---|---|---|---|---|---
Unigram Baseline | 0.2292 | 0.7184 | 0.1821 | 0.3669 | 0.1543 | 0.2420
Reformer | 0.6199 | 0.8941 | 0.6047 | 0.7817 | 0.5848 | 0.6953
Transformer XL | 0.6955 | 0.9174 | 0.6856 | 0.8437 | 0.6674 | 0.7738
GPT: Beam Search | 0.4867 | 0.8247 | 0.4639 | 0.6530 | 0.4424 | 0.5587
GPT2: Beam Search | 0.6250 | 0.8822 | 0.6109 | 0.7767 | 0.5935 | 0.7007
Table 6: Results on the Switchboard Corpus with 10% noise present, averaged
across all character positions


#  Elevater: A Benchmark and Toolkit for Evaluating Language-Augmented Visual
Models

Chunyuan Li∗1♠, Haotian Liu∗2, Liunian Harold Li3, Pengchuan Zhang1, Jyoti
Aneja1
Jianwei Yang1, Ping Jin1, Yong Jae Lee2, Houdong Hu1, Zicheng Liu1, Jianfeng
Gao1
1Microsoft 2University of Wisconsin–Madison 3UCLA

###### Abstract

Learning visual representations from natural language supervision has recently
shown great promise in a number of pioneering works. In general, these
language-augmented visual models demonstrate strong transferability to a
variety of datasets/tasks. However, it remains a challenge to evaluate the
transferablity of these foundation models due to the lack of easy-to-use
toolkits for fair benchmarking. To tackle this, we build Elevater††∗Equal
Technical Contribution ♠Project Lead111 Evaluation of Language-augmented
Visual Task-level Transfer, the first benchmark to compare and evaluate pre-
trained language-augmented visual models. Several highlights include: $(i)$
Datasets. As downstream evaluation suites, it consists of 20 image
classification datasets and 35 object detection datasets, each of which is
augmented with external knowledge. $(ii)$ Toolkit. An automatic hyper-
parameter tuning toolkit is developed to ensure the fairness in model
adaption. To leverage the full power of language-augmented visual models,
novel language-aware initialization methods are proposed to significantly
improve the adaption performance. $(iii)$ Metrics. A variety of evaluation
metrics are used, including sample-efficiency (zero-shot and few-shot) and
parameter-efficiency (linear probing and full model fine-tuning). We will
release our toolkit and evaluation platforms for the research community.

##  1 Introduction

Visual recognition has become ubiquitous in our society [60], with
applications in geo-localization [51], action recognition [57], street number
transcription [44], satellite remote sensing [22], medical imaging [62], self-
driving cars [19], etc. Core to many of these applications are visual
recognition tasks such as image classification (IC) and object detection (OD).
It is of high value to develop transferable visual models that perform well on
a wide range of downstream application scenarios. By leveraging large web
crawled image-text corpora, recent advances in language-augmented visual
models such as CLIP [51] and ALIGN [24] have demonstrated strong transfer
performance, making this direction arguably the most practical visual learning
approach, because $(i)$ open-set recognition is enabled due to the problem
reformulation from classification to retrieval; $(ii)$ generalization is
improved since the form of language supervision allows higher coverage of
visual concepts.

The success has quickly inspired many follow-up large-scale pre-training works
[68, 72, 69, 36, 43, 20, 34, 74]. Each of them developed their own evaluation
experiments, covering a customized subset of tasks, and leaving the details of
model adaptation process less accessible. These tasks are often studied
repeatedly in isolation, and require a lot of development effort to work. To
fill this gap, in this paper we develop an open-resource benchmark and
toolkit, designed to make working on task-level transfer easier, more
vigorous, and reproducible. We provide software to evaluate models without
task-specific tuning, and also comprehensive metrics to measure model
performance from various perspectives.

Our main findings and contributions can be summarized as follows:

  * •

Benchmark: Datasets and Knowledge. We build the first publicly available
benchmark to evaluate the recent language-augmented visual models. It includes
a suite of 20 and 35 datasets for image classification (IC) and object
detection (OD), respectively. Beyond that, external knowledge is collected for
each dataset to further augment language, including WordNet hierarchy [42],
WordNet definition [42], Wiktionary definition [41], and GPT-3 knowledge [4],
which represents the first comprehensive benchmark in evaluating large-scale
task-level transfer.

  * •

Comprehensive Metrics. To take into account the practical efficiency in the
deployment stage in the real world, a variety of evaluation metrics are
developed with the full sample and parameter spectrum in mind. Specifically,
we include sample-efficiency (zero-shot, few-shot and full-shot) and
parameter-efficiency (linear probing and full model fine-tuning) in
experiments.

  * •

Reproducible Toolkit & Language-augmented Adaptation Methods. We develop an
open-source software toolkit to support model adaptation and evaluation.
Automatic hyper-parameter tuning is employed to avoid human-in-the-loop
tuning, thus reducing human labor and ensuring fair comparisons of different
model checkpoints. We propose new model adaptation methods to leverage the
full power of pre-trained language-augmented visual models, which
significantly outperform the traditional vision-only adaptation method in all
settings. All these methods are implemented as strong baselines, serving as a
starting tutorial to encourage more advanced adaptation solutions.

  * •

Novel Empirical Findings. Several of our observations challenge previously
reported results. $(i)$ Random initialization of the linear head in language-
augmented visual models is sub-optimal; instead, leveraging both text and
vision in these models consistently yields better performance. Importantly,
few-shot always performs better than zero-shot, in contrast to the results
reported in [51]. $(ii)$ For language-augmented visual models, linear probing
consistently performs better than full model fine-tuning in the data-limited
settings. As the training dataset size increases, fine-tuning gradually
overtakes linear probing. In contrast, for traditional language-free models,
fine-tuning always performs better. $(iii)$ External knowledge can benefit
zero-shot and few-shot learning performance, when used properly. Further, the
implicit knowledge generated by GPT3 [4] improves the zero-shot performance.

##  2 Related Work

###  2.1 Visual Recognition Benchmarks: Zero-shot and Transfer Learning

Zero-shot learning in computer vision has been studied for decades. The
research topic has witnessed two types of distinctive notions of zero-shot:
the traditional class-level zero-shot that usually refers to the study of
generalizing to unseen object categories [30], and the recently popular task-
level zero-shot that refers to the study of generalizing to unseen
datasets/tasks [32, 51]. In Table 1, we compare our benchmark against existing
benchmarks. Existing zero-shot learning benchmarks are developed for the
class-level zero-shot setting. They are usually in single domain, with a
manual split of categories to product disjoint training and test categories,
_e.g.,_ Animal with Attributes (AwA) [31], Caltech-UCSD Birds-200 (CUB) [64],
SUN [47], aPY [14], and ZS-ImageNet [52, 17].

In contrast, our benchmark focuses on task-level transfer across domains,
_i.e.,_ it aims to evaluate the transferability of models, by pre-training
from their own large corpus, then evaluating zero-shot performance on a
diverse set of downstream datasets. This setting has been recently studied
[32, 51, 33, 72], and is arguably more practical for real-world applications,
as it brings the convenience towards the spirit of one-model-for-all. The
well-known ImageNet-1K dataset [9] was originally proposed as a large dataset
for model training and testing. It has also recently been considered as one
downstream task to study zero-shot transfer [51, 33, 72]. Our work presents
the first _public benchmark_ to standardize the zero-shot task-level transfer
setting. Note that visual task transfer has been previously explored in VTAB
[73], which measures good visual representations as those that adapt to
diverse, unseen tasks with an emphasis on few examples. The pre-trained models
and task adaptation in VTAB are considered for vision backbones only, and no
language model/modality is involved. Our benchmark shares a similar spirit of
task-level transfer to VTAB, but strives for analyzing the vital importance of
language and knowledge in visual transfer. We have further made several novel
contributions to consolidate the benchmark: $(i)$ We add external knowledge
for each dataset to cultivate new research directions in knowledge-augmented
visual models, inspired by the success of knowledge in traditional class-level
transfer. $(ii)$ We consider the full spectrum in measuring the sample-
efficiency of task adaptation, including zero-shot, few-shot, and full-shot.

| Benchmark Statistics | Evaluation Settings |
---|---|---|---
Problem |  | #Datasets | #Image | #Concepts | Knowledge Source | Zero | Few | Full
IC |  AwA [31] | 1 | 30337 / 6985 | 40 / 10 | Attributes | $\checkmark$ |  |
CUB [64] | 1 | 8855 / 2933 | 150 / 50 | Attributes | $\checkmark$ |  |
SUN [47] | 1 | 12900 / 1440 | 645 / 72 | Attributes | $\checkmark$ |  |
aPY [14] | 1 | 12695 / 2644 | 20 / 12 | Attributes | $\checkmark$ |  |
ZS-ImageNet [52] | 1 | 1.2M / 54K | 1K / 360 | WordNet | $\checkmark$ |  |
|  ImageNet-1K [9] | 1 | 1.2M / 50K | 1K | WordNet | $\checkmark$ |  | $\checkmark$
|  VTAB [73] | 19 | 2.2M / - | 940 | - |  | $\checkmark$ | $\checkmark$
| Elevater (Ours) | 20 | 638K / 193K | 1151♢ | WordNet, Wiki, GPT-3 | $\checkmark$ | $\checkmark$ | $\checkmark$
OD |  LVIS [21] | 1 | 120k / 40K | 1723 | WordNet |  |  | $\checkmark$
Elevater (Ours) | 35 | 132K / 20K | 314♢ | WordNet, Wiki, GPT-3 | $\checkmark$ | $\checkmark$ | $\checkmark$

Table 1: Comparison of dataset statistics and evaluation settings. For
existing zero-shot datasets in IC, the numbers of images and concepts are
reported for development and evaluation stages separately. ♢ It represents the
total #concepts in the benchmark to evaluate task-level transfer, and there is
no train-evaluation category split as in class-level transfer.

###  2.2 Pre-trained Models in NLP: Benchmarks, Adaptation, and Knowledge

Our work takes major inspiration from the development of pre-trained language
models in natural language processing (NLP) in several aspects: $(i)$
Benchmarks. Platforms with a suite of small datasets such as GLUE
[66]/SuperGLUE [65] have been extensively used to evaluate the general
language understanding ability of pre-trained models [11]. Recently, there is
a trend in NLP to develop task-agnostic models such as the GPT family [4] that
demonstrate task-level transfer learning ability, enabling zero-shot and few-
shot transfer to downstream datasets. The success in NLP encourages us to
build a generic benchmark to measure the similar transferability for visual
models. $(ii)$ Efficient adaptation. The democratization of large pre-trained
models for efficient adaptation in downstream applications is an important
topic in practice. Many algorithms have been developed for various efficiency
considerations, including adapters [23] and prompt tuning [35, 37]. It has
inspired a few recent works for language-augmented visual models [75, 59, 70,
18]. Our benchmark can serve as a comprehensive playground to quantify the
progress in the emerging field of visual model adaptation. We also propose a
novel language-initialized adaptation method as a strong baseline. $(iii)$
Knowledge. Knowledge-intensive tasks [40, 49] — those where a human can only
be expected to perform the task with access to a knowledge source such as
Wikipedia — are challenging for even cutting edge NLP and vision-and-language
models, as it is infeasible to train large models to memorize everything. KILT
[49] is a benchmark that contains a suite of tasks/datasets for evaluating and
analyzing knowledge-intensive NLP models. Similarly, we also add various
external knowledge sources in each downstream dataset for our vision
benchmark.

##  3 Benchmarks

###  3.1 A Suite of Datasets with Language/Knowledge Augmentations

As a proxy for performing unseen tasks in the wild, we collect a diverse set
of public datasets from various domains in computer vision, as the basis of
our benchmark. Specifically, we consider 20 datasets for IC and 35 datasets
for OD. We exhibit the dataset names in Figure 1 (a), and the detailed
statistics of each dataset in the Appendix. It is recommended in [51] that
studying task-level zero-shot transfer is a way of measuring the task learning
capabilities of machine learning systems. The task definition of each
downstream recognition dataset is typically specified using category names.
Adding customer specification/note is a natural way to clarify the task
definition _e.g.,_ the attribute or explanation of a visual concept.
Importantly, a similar spirit has been implemented in traditional class-level
zero-shot by adding individual domain-specific knowledge (see Table 1), and
demonstrated promising zero-shot performance gains. In this paper, we
generalize the terms to task-level, collecting external knowledge from general
sources for our benchmark.

  * •

WordNet Hierarchy (def_path). The words along the traversal path from the
query node in WordNet [42] to the highest parent node is recorded as the
hierarchy knowledge.

  * •

WordNet Definition (def_wn). The definition in WordNet synsets [42] is used to
explain the query.

  * •

Wiktionary Definition (def_wik). The definition of a query in Wiktionary [41]
is used.

  * •

GPT3 Knowledge (gpt3). For the above three knowledge sources, it is not always
feasible to retrieve valid knowledge for any query. To enable full knowledge
coverage, we propose to use GPT3 [4] to generate “pseudo” knowledge, where the
in-context-learning prompt is constructed with multiple pairs of class names
and their Wiktionary definitions. We generate five GPT3 knowledge sequences
for each class name, by constructing different context prompts with randomly
sampled pairs.

|
---|---

(a) Dataset names. The font size is proportional to the number of concepts in
each dataset.

|
---|---

(b) Examples of collected external knowledge.

Figure 1: Illustration of our benchmark. Left: Image classification, Right:
Object detection.

In Fig. 1 (b), we show examples to illustrate the knowledge sources. In
practice, there is a trade-off between the knowledge quality and its coverage.
For example, WordNet has relatively rich and precise knowledge, but the
coverage is low; GPT3 knowledge has the full coverage (as it is generated via
prompting a pre-trained neural language model), but it is hard to assess its
quality. In the experiment section, we provide baseline results to demonstrate
the benefits of external knowledge, and encourage the community to design
advanced modeling/prompting techniques to leverage these knowledge sources.

###  3.2 Pre-trained Models for Transfer Learning

#### Industry Track and Academic Track.

Our benchmark is essentially an evaluation platform for pre-trained models,
whose performance largely depends on the scale of the pre-training corpus.
Larger corpus typically yields higher performance, but unfortunately results
in a barrier to many participants, especially a majority of researchers from
university labs. To increase inclusivity, we create two tracks with
restrictions on the pre-training data scale: $(i)$ Industry track has no limit
on pre-training data scale, except that images in our benchmark are not
allowed in pre-training when reporting zero-/few-shot performance. This track
aims to explore the scaling limit. $(ii)$ Academic track is a setting that
limits the data in established public large datasets (_i.e.,_ ImageNet-21K
[9], GCC3M [54] & 12M [5], YFCC15M [61]). This track is more academic-
friendly, aiming to encourage the exploration in data-efficient pre-training
methodologies. We encourage participants in this track to report the pre-
training datasets to enable reproducible research.

#### Image Classification

We consider four checkpoints to produce baseline results for IC. The first two
are for the industry track, the last two are for the academic track.

  * •

Supervised ViT-B32  [12] represents a checkpoint for the traditional language-
free visual models, where model training is performed on ImageNet-22K with
cross-entropy loss.

  * •

CLIP ViT-B32 [51] represents a checkpoint for the family of the language-
augmented visual models, trained with 400M image-text pairs.

  * •

UniCL Swin [68] represents knowledge-free language-augmented visual models
with Swin [38] as the visual backbone, trained in the academic setting with
ImageNet-21K, which excludes ImageNet-1K categories from ImageNet-22K.

  * •

Knowledge UniCL Swin [55] represents knowledge-enriched language-augmented
visual models. Its pre-training setting is the same as UniCL Swin, but
external knowledge such as Wiktionary is leveraged in model pre-training.

#### Object Detection

We consider four checkpoints to produce baseline results for OD. They are for
the academic track, as they are pre-trained on public datasets. All of them
employ Swin-Tiny backbone [38].

  * •

DyHead [8] represents a checkpoint for the traditional language-free object
detector, where model is pre-trained on Object365 [53] without leveraging the
category name information.

  * •

GLIP [34] represents a checkpoint for the family of the language-augmented
object detector, trained with Object365 and Flicker phrase grounding data
[50].

  * •

GLIP-A [34] represents knowledge-free language-augmented object detector,
where model is trained on Object365 and the semantics of category names is
leveraged.

  * •

Knowledge GLIP-A [55] represents knowledge-enriched language-augmented object
detector. Its training setting is the same as GLIP-A, except that Wiktionary
knowledge is leveraged in model pre-training.

In summary, among four checkpoints for each problem, the first two are used to
compare the state-of-the-art in language-free and language-augmented models,
and latter two are used to compare the knowledge-free and knowledge-augmented
models (both belongs to language-augmented models, as knowledge is presented
as a structured form of language).

###  3.3 Evaluation Settings: Efficiency Considerations

#### Sample-efficiency: Zero-shot, Few-shot, and Full-shot.

Due to the high cost of annotating data, it is often desired to provide a
small number of labeled image-label pairs for model adaptation in a downstream
transfer dataset. Good models should be able to reach high performance in this
data-limited scenario. To assess this ability, we construct a variety of
settings, by varying the number of training set size $N$ in the downstream
dataset. For IC, $N=0,5,20,50,|\mathcal{D}|$, where $|\mathcal{D}|$ is the
total number of samples for a given dataset. For OD,
$N=0,1,2,5,10,|\mathcal{D}|$. Three random seeds are chosen, each of which
identifies a subset of samples from the full dataset in a deterministic
manner. In other words, once the random seed is given, the indices of training
samples in few-shot settings are fixed to encourage reproducible research.

#### Parameter-efficiency: Linear Probing vs Full Model Fine-tuning.

Maintaining a small number of dataset-specific model parameters is often
favored for model maintenance, as it can be expensive to maintain a unique
copy of large model checkpoints for each of the thousands of downstream AI
applications. In IC, linear probing provides a simple strategy for training a
dataset-specific linear embedding matrix, while keeping the pre-trained visual
backbone frozen. It arguably represents the minimum cost solution for
parameter-efficient model adaptation. In contrast, fine-tuning often updates
the entire weights in backbone and linear head, representing the most
expensive solution to model adaptation. We implement these two representative
solutions as baselines. In OD [67], linear probing typically means updating
the linear heads for classification and localization (including centers) tasks
only, while fine-tuning means update all model weights including the backbone
and the detectors.

##  4 Toolkits

###  4.1 Automatic Hyper-parameter Tuning

#### Image Classification.

For a given dataset, we split its training set into training and validation
with a ratio 80% vs 20%. At least one training sample per class is ensured for
training and validation. Grid search is applied over learning rate $\eta$ and
weight decay $\alpha$. In the hyper-parameter search stage, the model is
trained with a given configuration $(\eta,\alpha)$ for 10 epochs, the best
hyper-parameter configuration is chosen as the one with the best validation
performance along the entire process. After that, a final run is performed for
50 epochs to report the performance on the testing set.

#### Object Detection.

A validation set is chosen in the hyper-parameter search stage. We consider
validation set size $(1,1,1,3,full)$ for $N=1,2,5,10,|\mathcal{D}|$,
respectively. For each type of checkpoints (DyHead, GLIP) and each adaption
method, we have a set of pre-selected hyparameters, _i.e.,_ batch size
|$\mathcal{B}$|, initial learning rate $\eta_{0}$ and weight decay $\alpha$,
as shown in Table B in Appendix. They are determined by either empirical rules
or simple hyper-parameter tuning. For each setting and each train/val split,
we evaluate on the val split after every training epoch to decrease the
learning rate in a step-wise manner. More specifically, we use the PyTorch
ReduceLROnPlateau with patience 3 and factor 0.1 to decrease the learning rate
when there is no improvement on val. We terminate the fine-tuning process if
we do not see improvements for continuously 9 epochs, return the checkpoint
with the best score on val, and report its score on the test split. For each
few-shot setting, we random sample the train/val split 3 times, and report the
average score and standard deviation on the test split.

###  4.2 Model Adaptation Methods

We provide several model adaptation methods as strong baselines, which allow
effective transfer learning of pre-trained visual models. The ideas are
illustrated in Figure 2. For a downstream dataset, we first represent it in a
triplet-wise data format
$\mathcal{D}=\\{(\boldsymbol{x}_{n},{\boldsymbol{t}}_{n},y_{n})\\}_{n=1}^{N}$,
where $\boldsymbol{x}\in\mathcal{X}$ is the image,
${\boldsymbol{t}}\in\mathcal{T}$ is its corresponding language description,
and $y\in\mathcal{Y}$ is a label indicating the index of the grouped or unique
language description in the dataset. In IC, the number of labels
$|\mathcal{Y}|=K$, the number of category names.

Figure 2: Illustrative comparison among different model evaluation and
adaption methods.

#### Language-free Visual Models.

Most existing visual models are language-free, as language information is
often not considered in training, _e.g.,_ supervised and self-supervised
learning methods. Such models can not be directly used for zero-shot transfer,
and model adaptation is often enabled by adding additional weights. For each
image $\boldsymbol{x}$, an image encoder model $f_{\boldsymbol{\theta}}$
parameterized by $\boldsymbol{\theta}$ first represents $\boldsymbol{x}$ as a
visual feature vector ${\boldsymbol{h}}\in\mathbb{R}^{D\times 1}$:
${\boldsymbol{h}}=f_{\boldsymbol{\theta}}(\boldsymbol{x})$. A linear
classifier with parameters ${{\bf W}}\in\mathbb{R}^{D\times K}$ (we absorb the
bias ${\boldsymbol{b}}$ in ${{\bf W}}$ for simplicity) is used, where ${{\bf
W}}$ is randomly initialized, shown in Figure 2 (a).

#### Language-augmented Visual Models.

Recent works [51, 24] that learn visual models with language supervision often
employ a two-encoder architecture. Besides the image encoder model
$f_{\boldsymbol{\theta}}$, a text encoder
$f_{\boldsymbol{\phi}}({\boldsymbol{t}})$ parameterized by $\boldsymbol{\phi}$
is also used to encode text ${\boldsymbol{t}}$. Additional projection layers
${{\bf W}}_{u}$ and ${{\bf W}}_{v}$ are introduced for image and language
features, embedding them into a joint space with dimension $P$, with projected
features as $\boldsymbol{u}$ and $\boldsymbol{v}$ respectively. As shown in
Figure 2 (b), zero-shot learning can be directly performed in this space: the
mean text feature $\boldsymbol{v}$ is first obtained for each category, by
averaging text features of the category name in different language prompts.
The image is predicted as the category yielding the highest similarity
$\boldsymbol{u}^{\top}\boldsymbol{v}$.

  * •

Random initialized Adaptation. In the original CLIP paper [51], to enable
model adaptation, a linear classifier with parameters ${{\bf
W}}\in\mathbb{R}^{P\times K}$ (similarly, we absorb the bias
${\boldsymbol{b}}$ in ${{\bf W}}$ for simplicity) is added on the projected
features, where ${{\bf W}}$ is randomly initialized, shown in Figure 2 (c)
This method often provides good linear probing results, but much worse fine-
tuning performance, a surprisingly counter-intuitive observation we have found
in our experiments.

  * •

Language-initialized Adaptation. We argue that the above issue is caused by
the fact that the full capacity of language-augmented visual models is not
leveraged. The power of pre-trained language encoder and text inputs must play
a vital role in model adaptation. Hence, we propose two language-initialized
adaption methods, each of which is ensured as a fair comparison variant for
language-augmented and language-free models, respectively. $(i)$ Separate-
Head. For the linear head ${{\bf W}}\in\mathbb{R}^{P\times K}$ added on the
projection space, we initialize ${{\bf W}}$ with ${{\bf V}}$ (bias terms are
initialized as zeros), shown in Figure 2 (d). In this way, the visual and text
heads are separated. $(ii)$ Merge-Head. To fairly compare with the language-
free model adaptation, the linear head should directly be added on the
backbone (before the projection layer) to ensure that the same number of
trainable parameters are updated. Therefore, we propose to initialize ${{\bf
W}}\in\mathbb{R}^{D\times K}$ in this case with the multiplication result of
two linear matrices ${{\bf W}}_{v}{{\bf V}}$, shown in Figure 2 (e).

#### Discussion.

We highly recommend the proposed language-initialized methods as the standard
for the community to adapt the pre-trained language-augmented visual models in
their downstream tasks for two reasons: $(i)$ This simple method yields
surprisingly superior empirical performance, as comprehensively demonstrated
in our experiments. It significantly improves the default random
initialization method for both linear probing and full model fine-tuning,
especially in the few-shot settings. $(ii)$ It provides an effective mechanism
to leverage the external knowledge that is collected for a downstream task in
our benchmark. As the knowledge can be concatenated with the original language
prompt, then encoded into contextualized text features. In contrast, random
initialization would ignore this knowledge source. We hope the proposed
language-initialized method can serve as a strong baseline to encourage the
community to further build more efficient adaptation methods.

In OD, GLIP is a language-augmented detector, whose overall architecture can
be simply considered as adding a cross-modal module over the CLIP-like dual-
encoder. In GLIP, its linear probing has been implemented via updating ${{\bf
W}}_{v}$ and ${{\bf W}}_{t}$ in [34]. A prompt-tuning strategy was proposed,
by initializing the language input of the cross-modal module as ${{\bf V}}$,
and only updating ${{\bf V}}$ in adaptation. This is similar to our language-
init strategy. From this perspective, the boundary of linear probing and
prompt tuning becomes blurred, with the major difference in if there exist a
cross-modal module.

##  5 Baseline Results

In this section, we present the major experimental results with our benchmark
to address two research questions. Q1: The importance of language in visual
model transfer in adaptation stage. Q2: We identify three playgrounds that our
benchmark can help to cultivate research, including sample-efficiency,
parameter-efficiency and external knowledge for visual transfer. We also
present novel empirical findings as they appear in the specific setting.

#### Experiment settings

To make a fair comparison between different methods, we conduct experiments
with FP32 precision. Our preliminary experiments show that on average FP16 and
FP32 yields similar zero-shot performance, while FP32 models outperform FP16
ones on 16 out of 20 datasets.

Settings | Init Method | 20 IC datasets |
---|---|---|---
Checkpoint | Adaptation | Zero-shot | Few-shot (5, 20, 50) | Full
Industry Track (No pre-train data scale limit) |
CLIP  (ViT-B32) | Linear Probing | Random | 56.64 |  58.09 $\pm$ 2.80, 69.97 $\pm$ 1.30, 74.09 $\pm$ 0.69 | 77.89
Linear Probing | Language-S |  65.35 $\pm$ 1.24, 71.69 $\pm$ 0.93, 74.89 $\pm$ 0.79 | 78.37
Linear Probing | Language-M |  65.88 $\pm$ 0.79, 72.05 $\pm$ 0.85, 75.08 $\pm$ 0.73 | 79.02
Fine-tuning | Random |  29.75 $\pm$ 6.64, 46.76 $\pm$ 11.9, 61.70 $\pm$ 9.97 | 77.77
Fine-tuning | Language-S |  63.29 $\pm$ 3.18, 72.19 $\pm$ 1.31, 75.70 $\pm$ 1.14 | 80.17
Supervised  (ViT-B32) | Linear Probing | Random | - |  51.74 $\pm$ 2.67, 61.98 $\pm$ 1.66, 66.41 $\pm$ 1.17 | 71.11
Fine-tuning | Random |  51.84 $\pm$ 2.58, 69.10 $\pm$ 1.25, 73.72 $\pm$ 1.42 | 78.94
Academic Track (Pre-trained on large established public datasets) |
UniCL  (Swin-Tiny) | Linear Probing | Language-S | 27.15 |  54.31 $\pm$ 4.15, 66.42 $\pm$ 2.08, 70.49 $\pm$ 1.01 | 74.75
Fine-tuning | Language-S |  44.75 $\pm$ 5.42, 56.53 $\pm$ 5.37, 67.90 $\pm$ 5.31 | 78.48
Knowledge UniCL  (Swin-Tiny) | Linear Probing | Language-S | 33.44 |  55.06 $\pm$ 2.36, 66.26 $\pm$ 1.56, 70.16 $\pm$ 1.09 | 74.47
Fine-tuning | Language-S |  48.41 $\pm$ 2.84, 58.06 $\pm$ 4.30, 71.66 $\pm$ 2.02 | 78.05

Table 2: Averaged results on 20 IC datasets. For model adaptation with the
language-augmented initialization methods, Language-S and Language-M indicates
the separate-head initialization in Figure 2 (d) and merge-head initialization
in Figure 2 (e), respectively.

Settings | 35 OD datasets
---|---
Checkpoint | Adaptation | Zero-shot | Few-shot (1, 3, 5, 10) | Full
GLIP  (Swin-Tiny) | Prompt | 19.7 |  29.7$\pm$ 0.4, 36.5$\pm$ 0.6, 39.0$\pm$ 1.1, 41.8$\pm$ 1.2 | 54.4
Linear Probing |  22.2$\pm$ 0.1, 24.4$\pm$ 0.2, 25.1$\pm$ 0.2, 25.6$\pm$ 0.6 | 35.2
Fine-tuning |  32.2$\pm$ 0.7, 39.2$\pm$ 0.3, 42.5$\pm$ 0.9, 49.1$\pm$ 0.6 | 63.2
DyHead  (Swin-Tiny) | Linear Probing | - |  15.2$\pm$ 0.6, 19.2$\pm$ 0.9, 19.8$\pm$ 1.0, 20.6$\pm$ 1.1 | 31.4
Fine-tuning |  25.6$\pm$ 0.4, 37.1$\pm$ 0.5, 40.1$\pm$ 1.5, 44.6$\pm$ 0.7 | 63.9

Table 3: Averaged results on 35 OD datasets.

###  5.1 The Role of Language for Vision

#### Effectiveness of Language-initialized Adaptation Methods.

In Table 5, we benchmark and compare the effectiveness of the proposed
language-initialization methods with the checkpoint CLIP ViT-B32. The Merge-
Head scheme is consistently better than Separate-Head scheme in all settings
(though the gain is minor). This is because the latter often have more
parameters than the former, as $P=512<D=768$. To ensure fair comparisons with
the random initialization of linear head in CLIP [51] (# trainable parameters
is the same), we consider the Separate-Head language-initialization scheme as
default, unless the Merge-Head scheme is specified.

Figure 3: Comparison between random- and language- initialized adaptation.

As shown in Fig. 3, under both linear probe (LP) and fine-tuning (FT)
settings, language-based initialization significantly outperforms random
initialization in all settings. Notably, we show that even with very few shots
(_e.g.,_ 2-shots), both our LP and FT is able to outperform the zero-shot
CLIP. This is contradictory to the finding in the original CLIP paper [51],
where zero-shot outperforms linear probing in the fewer shot (less than 4)
settings. With the proposed language-init method, one can ensure that few-shot
performance is always better than zero-shot, as we essentially reduce to zero-
shot when zero iteration is updated in our language-init method. Moreover, we
also find that with the random initialization, FT performs significantly worse
than LP under few-shot settings. However, with the language-init, FT starts to
outperform LP with more than 20 shots. Both findings demonstrate the proposed
language-based initialization is consistently effective, suggesting that it is
a useful and important technique, and should be the standard adaptation method
for the recent language-augmented visual models like CLIP.

#### The Competition of Foundation Models: Language-free vs Language-
augmented.

We argue that the correct adaptation methods for language-augmented visual
models should leverage both the pre-trained visual and text encoder. It is not
sufficient to solely transfer from the visual encoder (though the model size
is comparable with language-free counterpart), the pre-trained language
encoder plays an important role in task transfer. In Table 5, with language-
init adaptation methods, we now see that the language-augmented model (CLIP)
consistently outperforms language-free model (Supervised ViT) in all settings,
especially for the limited data settings. Further, language-augmented models
are able to perform zero-shot task transfer, while traditional language-free
models cannot. Similar conclusions can be drawn for OD in Table 5. Hence, we
recommend the use of language-augmented visual models for task-level transfer.

|
---|---
(a) IC: Sample Efficiency | (b) OD: Sample Efficiency
|
(c) IC: Parameter Efficiency | (d) OD: Parameter Efficiency

Figure 4: Adaptation efficiency considerations. For IC, comparison of
adaptation efficiency between CLIP and supervised ViT (Sup-ViT). For OD,
comparison of adaptation efficiency between GLIP and DyHead.

###  5.2 Playground I: Sample Efficiency

We explore the sample efficiency in Fig. 4 (a) for IC. First, we find that
CLIP-based adaptation consistently outperforms supervised ViT (Sup-ViT)
models, and yields a significant 13% gain in 5-shot settings. Even without any
training samples (zero-shot), CLIP outperforms the 5-shot Sup-ViT by 5% gain.
This suggests that CLIP models allow a better sample efficiency than
supervised ViT models. With more training samples, the performance gap between
the two checkpoints becomes smaller, _e.g.,_ around 1% with the full data.
Furthermore, we find that fine-tuning CLIP yields better performance than
linear probing in >20-shot settings, while being worse in the 5-shot setting.
This is a bit surprising, as it is contradictory to the common convention that
fine-tuning is always better than linear probing. We hypothesize this is
because fine-tuning tends to over-fit in the scenarios with a large number of
trainable parameters and a small number of training samples. Overall, it
suggests that fine-tuning CLIP potentially has a better sample efficiency than
linear probing, and a better adaptation strategy on fewer-shot settings can be
explored in the future. For supervised ViT, FT is always better than LP, the
performance gap becomes larger when more samples are used. In 5-shot settings,
the gap is minor, which is similar to observations made for supervised CNNs
[71, 25, 26].

As for the comparison of foundation models, we suggest to report the
evaluation results on the entire spectrum of sample-efficiency to fully study
the behaviors of a pre-trained model. If compute resource is limited, zero-
shot or few-shot evaluation can be used as quick assessment.

We explore the sample efficiency for OD in Fig. 4 (b). The conclusion is
similar to IC in that language-augmented visual model (_i.e.,_ GLIP) is more
sample-efficient than language-free visual model (_i.e.,_ DyHead), when the
models are adapted using either LP or FT settings. The performance gap is
large in the fewer-shot settings and small in the full-shot settings. The
difference is that FT consistently outperforms LP in all settings for OD.

###  5.3 Playground II: Parameter Efficiency

We study the parameter efficiency in Fig. 4 (c) for IC. For CLIP, we
experiment two different settings of linear probing CLIP on whether to merge
the last two linear projection layers ($\mathbf{W}_{v}$ and $\mathbf{V}$ in
Fig. 2). Merging these two layers in CLIP allows 1.5$\times$ trainable
parameters as keeping them separated. First, it generally shows the trend that
a larger number of trainable parameters leads to a better performance, as
demonstrated by three curves/scenarios: full-shot CLIP, full-shot Sup-ViT and
5-shot FT. It verifies that LP and FT provide the lower bound and upper bound,
in terms of both #parameter and performance. Most existing parameter-efficient
adaptation methods play a trade-off game. However, in the scenario of 5-shot
CLIP, we do notice a slight drop in performance when we further increase the
number of trainable parameters to full-model fine-tuning. It suggests that the
scenario of adapting language-augmented visual models for data-limited
settings is a much more meaningful battle field to explore the line of
research in parameter-efficient adaptation methods, as the best performance
may potentially require an optimal number of trainable parameters, which is
less explored so far.

We study the parameter efficiency in Fig. 4 (d) for OD. The overall trend is
similar in that better performance comes with more parameters. It turns out
that prompt tuning an language-augmented OD model is an effective parameter-
efficient approach. For example, prompting is better than linear probing in
GLIP. Further, prompt tuning GLIP outperforms fine-tuning DyHead in the 1-shot
setting, where the former has less than 0.1% parameters of the latter.

###  5.4 Playground III: The Benefit of External Knowledge for Vision

We investigate the effectiveness of external knowledge in Fig. 5, measured by
zero-shot task transfer performance. First of all, we find that leveraging
external knowledge in language-augmented visual models improves upon the
knowledge-free counterparts. For example, UniCL is improved from 27.15 to
29.92~33.93, and GLIP-A is improved from 11.53 to 11.70~13.30. Further, for
GPT3 knowledge, a larger number of generated knowledge items often leads to
higher performance. When combining GPT3 knowledge with Wiktionary knowledge,
we see a further performance boost. The gain is consistent for IC, but not
consistent for OD with a varying number of GPT3 knowledge items.

In Table 5, we study the role of knowledge for task transfer in model
adaptation. Initializing the linear head using encoded features with knowledge
is an effective way to leverage the collected knowledge sources, especially
for the fewer-shot settings. We also find that it may be insufficient to only
use the explicit database knowledge sources such as Wiktionary to initialize
the linear head of knowledge UniCL, which sometimes results in instability and
lower performance. We hypothesize this is because the partial knowledge
coverage of Wiktionary leads to a situation that some classes are initialized
with knowledge, while some are not. It results in two separate embedding
clusters, making it difficult to optimize. To solve this, we propose to
initialize with both Wiktionary and GPT3 knowledge (which leads to a notable
improvement, _e.g.,_ 1.4% absolute averaged performance gain in 50-shot FT).

Problem | Model | Baseline | External Knowledge Sources in Evaluation
---|---|---|---
wn_path | wn_def | wiki_def | gpt3 |  wiki_def & gpt3
IC | UniCL | 27.15 | 30.68 | 29.92 | 33.44 | 33.73 | 33.95
OD | GLIP-A | 11.53 | 12.43 | 11.70 | 13.14 | 11.98 | 13.30

Figure 5: Zero-shot task transfer with various external knowledge sources in
the evaluation stage. In (a) and (b), a varying number of generated GPT-3
knowledge sequences is utilized for inference, and “Wiki-and-GPT3” indicates
both Wiktionary and GPT3 knowledge are used simultaneously. The bottom table
summarizes the prediction result for each knowledge source.

##  6 Conclusions

In this paper, we have presented Elevater, a platform to evaluate the recently
emerging language-augmented visual models for task-level transfer. It consists
of 20 image classification datasets and 35 object detection datasets. All of
them are collected from public domains, and are enriched with various external
knowledge sources to improve the language modality. We have developed open-
source toolkits with an auto hyper-parameter pipeline and novel language-
initialized adaptation methods to ensure easy utilization and fair
comparisons. Strong baseline results are produced from the toolkit to
cultivate research in a variety of topics, _e.g.,_ more transferable language-
augmented visual foundation models, advanced model adaption methods (sample-
efficiency and parameter-efficiency), and external knowledge for task-level
transfer. The question of how to design general-purpose task-level
transferable visual models remains largely unanswered. Given benchmarks and
tookits we have developed from the perspective of language-augmented visual
models, we believe that Elevater can provide fertile soil for addressing this
challenge.

## Acknowledgments

The authors gratefully acknowledge Pengcheng He for helpful discussions to
have a separate track dedicated for users from academia, Baolin Peng and and
Zhengyuan Yang for the inspirations of GPT3 to generate knowledge for dialogue
and OK-VQA tasks. The benchmark depends on publicly available datasets; we
acknowledge all the original authors who made their datasets public. Please
follow the original license of each dataset and keep this benchmark for
academic purposes.

## References

  * [1] FER 2013: Kaggle challenges in representation learning facial expression recognition.  https://www.kaggle.com/.
  * [2] Sven Bambach, Stefan Lee, David Crandall, and Chen Yu.  Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions.  In ICCV, 2015.
  * [3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In ECCV, 2014.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, 2020.
  * [5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In CVPR, 2021.
  * [6] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote sensing image scene classification: Benchmark and state of the art.  Proceedings of the IEEE, 2017.
  * [7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In CVPR, 2014.
  * [8] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang.  Dynamic head: Unifying object detection heads with attentions.  In CVPR, 2021.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [10] Li Deng.  The MNIST database of handwritten digit images for machine learning research.  IEEE signal processing magazine, 2012.
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (VOC) challenge.  IJCV, 2010.
  * [14] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth.  Describing objects by their attributes.  In CVPR, 2009.
  * [15] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories.  In CVPR workshop, 2004.
  * [16] Jannik Fritsch, Tobias Kuehnl, and Andreas Geiger.  A new performance measure and evaluation benchmark for road detection algorithms.  In ITSC. IEEE, 2013.
  * [17] Yanwei Fu and Leonid Sigal.  Semi-supervised vocabulary-informed learning.  In CVPR, 2016.
  * [18] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  CLIP-adapter: Better vision-language models with feature adapters.  arXiv preprint arXiv:2110.04544, 2021.
  * [19] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  IJRR, 2013.
  * [20] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Zero-shot detection via vision and language knowledge distillation.  arXiv preprint arXiv:2104.13921, 2021.
  * [21] Agrim Gupta, Piotr Dollar, and Ross Girshick.  LVIS: A dataset for large vocabulary instance segmentation.  In CVPR, 2019.
  * [22] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  EuroSat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [23] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, 2019.
  * [24] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  arXiv preprint arXiv:2102.05918, 2021.
  * [25] Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller.  Recognizing image style.  arXiv preprint arXiv:1311.3715, 2013.
  * [26] Nikolaos Karianakis, Zicheng Liu, Yinpeng Chen, and Stefano Soatto.  Reinforced temporal attention and split-rate transfer for depth-based person re-identification.  In ECCV, 2018.
  * [27] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  NeurIPS, 2020.
  * [28] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In ICCV workshops, 2013.
  * [29] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [30] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.  Learning to detect unseen object classes by between-class attribute transfer.  In CVPR, 2009.
  * [31] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.  Attribute-based classification for zero-shot visual object categorization.  PAMI, 2013.
  * [32] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten.  Learning visual N-grams from web data.  In ICCV, 2017.
  * [33] Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  arXiv preprint arXiv:2107.07651, 2021.
  * [34] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  CVPR, 2022.
  * [35] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  arXiv preprint arXiv:2101.00190, 2021.
  * [36] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  arXiv preprint arXiv:2110.05208, 2021.
  * [37] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  arXiv preprint arXiv:2107.13586, 2021.
  * [38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  ICCV, 2021.
  * [39] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [40] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  OK-VQA: A visual question answering benchmark requiring external knowledge.  In CVPR, 2019.
  * [41] Christian M Meyer and Iryna Gurevych.  Wiktionary: A new rival for expert-built lexicons? Exploring the possibilities of collaborative lexicography.  2012\.
  * [42] George A Miller.  WordNet: An electronic lexical database.  MIT press, 1998.
  * [43] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  SLIP: Self-supervision meets language-image pre-training.  arXiv preprint arXiv:2112.12750, 2021.
  * [44] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [45] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics & Image Processing. IEEE, 2008.
  * [46] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In CVPR, 2012.
  * [47] Genevieve Patterson and James Hays.  SUN attribute database: Discovering, annotating, and recognizing scene attributes.  In CVPR, 2012.
  * [48] Malte Pedersen, Joakim Bruslund Haurum, Rikke Gade, Thomas B. Moeslund, and Niels Madsen.  Detection of marine animals in a new underwater dataset with varying visibility.  In CVPR Workshops, 2019.
  * [49] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al.  KILT: a benchmark for knowledge intensive language tasks.  arXiv preprint arXiv:2009.02252, 2020.
  * [50] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In ICCV, 2015.
  * [51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [52] Marcus Rohrbach, Michael Stark, and Bernt Schiele.  Evaluating knowledge transfer and zero-shot learning in a large-scale setting.  In CVPR, 2011.
  * [53] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In ICCV, 2019.
  * [54] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [55] Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, and Jianfeng Gao.  K-LITE: Learning transferable visual models with external knowledge.  arXiv preprint, 2022.
  * [56] Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra.  PlantDoc: A dataset for visual plant disease detection, 2019.
  * [57] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [58] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.  The german traffic sign recognition benchmark: a multi-class classification competition.  In IJCNN, 2011.
  * [59] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  VL-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  CVPR, 2022.
  * [60] Richard Szeliski.  Computer vision: algorithms and applications.  Springer Science & Business Media, 2010.
  * [61] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  YFCC100M: The new data in multimedia research.  Communications of the ACM, 2016.
  * [62] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant CNNs for digital pathology.  June 2018.
  * [63] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant cnns for digital pathology.  In MICCAI, 2018.
  * [64] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The Caltech-UCSD birds-200-2011 dataset.  2011\.
  * [65] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  NeurIPS, 2019.
  * [66] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * [67] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu.  Frustratingly simple few-shot object detection.  arXiv preprint arXiv:2003.06957, 2020.
  * [68] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Lu Yuan, Ce Liu, and Jianfeng Gao.  UniCL: unified contrastive learning in image-text-label space.  CVPR, 2022.
  * [69] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  FILIP: Fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [70] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  CPT: Colorful prompt tuning for pre-trained vision-language models.  arXiv preprint arXiv:2109.11797, 2021.
  * [71] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  NIPS, 2014.
  * [72] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [73] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  arXiv preprint arXiv:1910.04867, 2019.
  * [74] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  RegionCLIP: Region-based language-image pretraining.  CVPR, 2022.
  * [75] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  arXiv preprint arXiv:2203.05557, 2022.

##  Appendix A Detailed Dataset Statistics in the Benchmark Suite

In Table 4 and Table 5, we list the basic statistics of 20 image
classification datasets and 35 object detection datasets in the benchmark.

Dataset | #Concepts | Train size | Test size | Evaluation metric | Source link |
---|---|---|---|---|---|---
Hateful Memes [27] | 2 | 8,500 | 500 | ROC AUC | Facebook |
PatchCamelyon [63] | 2 | 262,144 | 32,768 | Accuracy | Tensorflow |
Rendered-SST2 [51] | 2 | 6,920 | 1,821 | Accuracy | OpenAI |
KITTI Distance [16] | 4 | 6,347 | 711 | Accuracy | KITTI website |
FER 2013 [1] | 7 | 28,709 | 3,589 | Accuracy | Kaggle fer2013 |
CIFAR-10 [29] | 10 | 50,000 | 10,000 | Accuracy | Tensorflow |
EuroSAT [22] | 10 | 5,000 | 5,000 | Accuracy | Tensorflow |
MNIST [10] | 10 | 60,000 | 10,000 | Accuracy | Tensorflow |
VOC 2007 Classification [13] | 20 | 2,501 | 4,952 | 11-point mAP | VOC 2007 |
Oxford-IIIT Pets [46] | 37 | 3,680 | 3,669 | Mean-per-class | Oxford-IIIT Pets |
GTSRB [58] | 43 | 26,640 | 12,630 | Accuracy | GTSRB website |
Resisc-45 [6] | 45 | 3,150 | 25,200 | Accuracy | Tensorflow |
Describable Textures [7] | 47 | 1,880 | 1,880 | Accuracy | DTD website |
CIFAR-100 [29] | 100 | 50,000 | 10,000 | Accuracy | Tensorflow |
FGVC Aircraft (variants) [39] | 100 | 3,334 | 3,333 | Mean-per-class | FGVC website |
Food-101 [3] | 101 | 75,750 | 25,250 | Accuracy | Tensorflow |
Caltech-101 [15] | 102 | 3,060 | 6,084 | Mean-per-class | Tensorflow |
Oxford Flowers 102 [45] | 102 | 1,020 | 6,149 | Mean-per-class | Tensorflow |
Stanford Cars [28] | 196 | 8,144 | 8,041 | Accuracy | Stanford Cars |
Country-211 [51] | 211 | 31,650 | 21,100 | Accuracy | OpenAI |
Total | 1151 | 638429 | 192677 | – | – |

Table 4: Statistics of 20 datasets used in image classification.

Dataset | #Concepts | #Image | #Annotated Regions | Source link
---|---|---|---|---
Train | Test | Train | Test
CottontailRabbits | 1 | 1980 | 10 | 2070 | 11 | Roboflow
EgoHands(generic) [2] | 1 | 3840 | 480 | 12015 | 1514 | Roboflow
MountainDewCommercial | 1 | 17 | 1 | 453 | 32 | Roboflow
Packages | 1 | 19 | 3 | 31 | 5 | Roboflow
Raccoon | 1 | 150 | 17 | 164 | 20 | Roboflow
WildfireSmoke | 1 | 516 | 74 | 516 | 74 | Roboflow
Pistols | 1 | 2377 | 297 | 2728 | 358 | Roboflow
Pothole | 1 | 465 | 67 | 1256 | 154 | Roboflow
MaskWearing | 2 | 105 | 15 | 696 | 96 | Roboflow
NorthAmericaMushrooms | 2 | 41 | 5 | 67 | 9 | Roboflow
OxfordPets(species) [46] | 2 | 2523 | 358 | 2527 | 358 | Roboflow
PKLot640 | 2 | 8691 | 1242 | 497856 | 70684 | Roboflow
ThermalCheetah | 2 | 90 | 14 | 152 | 31 | Roboflow
ThermalDogsAndPeople | 2 | 142 | 20 | 181 | 27 | Roboflow
BCCD | 3 | 255 | 36 | 3450 | 471 | Roboflow
HardHatWorkers | 3 | 5069 | 1766 | 19455 | 6808 | Roboflow
ShellfishOpenImages | 3 | 407 | 58 | 859 | 116 | Roboflow
EgoHands(specific) | 4 | 3840 | 480 | 12015 | 1514 | Roboflow
AerialMaritimeDrone(large) | 5 | 52 | 7 | 873 | 78 | Roboflow
AerialMaritimeDrone(tiled) | 5 | 371 | 32 | 1237 | 98 | Roboflow
VehiclesOpenImages | 5 | 878 | 126 | 1676 | 258 | Roboflow
BrackishUnderwater [48] | 6 | 11739 | 1468 | 28518 | 3466 | Roboflow
Dice | 6 | 576 | 71 | 1439 | 225 | Roboflow
Aquarium | 7 | 448 | 63 | 3324 | 584 | Roboflow
DroneControl | 8 | 32688 | 4675 | 32734 | 4694 | Roboflow
WebsiteScreenshots | 8 | 1688 | 242 | 76820 | 10656 | Roboflow
SelfDrivingCar | 11 | 24000 | 3000 | 156730 | 19598 | Roboflow
ChessPieces | 13 | 202 | 29 | 2108 | 376 | Roboflow
UnoCards | 15 | 6295 | 899 | 18885 | 2697 | Roboflow
PascalVOC [13] | 20 | 13690 | 3422 | 31356 | 7835 | Roboflow
AmericanSignLanguageLetters | 26 | 1512 | 72 | 1512 | 72 | Roboflow
Plantdoc [56] | 30 | 2128 | 239 | 7629 | 454 | Roboflow
BoggleBoards | 36 | 285 | 35 | 5727 | 647 | Roboflow
OxfordPets(breed) | 37 | 2437 | 345 | 2441 | 345 | Roboflow
OpenPoetryVision | 43 | 2798 | 402 | 8392 | 1198 | Roboflow
Total | 314 | 132314 | 20070 | 937892 | 135563 | –

Table 5: Statistics of 35 datasets used in object detection. Box mAP is used
as the evaluation metric. Datasets are downloaded from Roboflow. For the
datasets without a citation, we refer to Roboflow links for the original
sources.

##  Appendix B Experiment Details

For each type of checkpoints (DyHead, GLIP) and each adaption method, we have
a set of pre-selected hyparameters, _i.e.,_ batch size |$\mathcal{B}$|,
initial learning rate $\eta_{0}$ and weight decay $\alpha$, as shown in Table
B. They are determined by either empirical rules or simple hyper-parameter
tuning.

Settings | 35 OD datasets
---|---
Checkpoint | Adaptation |  |$\mathcal{B}$| | $\eta_{0}$ | $\alpha$
GLIP  (Swin-Tiny) | Prompt | 4 | 0.05 | 0.25
Linear Probing | 0.0001 | 0.05
Fine-tuning | 0.0001 | 0.05
DyHead  (Swin-Tiny) | Linear Probing | 4 | 0.0001 | 0.05
Fine-tuning | 0.0001 | 0.05

Table 6: Pre-selected hyperparameters for OD datasets.