# Adaptive multilingual speech recognition with pretrained models

###### Abstract

Multilingual speech recognition with supervised learning has achieved great
results as reflected in recent research. With the development of pretraining
methods on audio and text data, it is imperative to transfer the knowledge
from unsupervised multilingual models to facilitate recognition, especially in
many languages with limited data. Our work investigated the effectiveness of
using two pretrained models for two modalities: wav2vec 2.0 for audio and
MBART50 for text, together with the adaptive weight techniques to massively
improve the recognition quality on the public datasets containing CommonVoice
and Europarl. Overall, we noticed an 44% improvement over purely supervised
learning, and more importantly, each technique provides a different
reinforcement in different languages. We also explore other possibilities to
potentially obtain the best model by slightly adding either depth or relative
attention to the architecture.

Index Terms: speech recognition, multilingual, transformer, lstm, weight
factorization, weight decomposition, pre-training, wav2vec, bart

##  1 Introduction

The sequence-to-sequence approach is widely used in speech recognition (SR)
nowadays [1, 2, 3], and many research works are dedicated to show that their
capabilities relying on a single architecture often match or are even better
than traditional hybrid or CTC systems with separately optimized components
[4, 3]. Moreover, this approach is also successfully realized in multilingual
speech recognition, allowing one model to contain the information of many
languages shared in one single body of neural networks [5].

Despite such promising potential, this approach inevitably requires abundant
training data because it combines acoustic models, language models and
alignment into the same model [6, 7]. In fact, the labeled data necessary for
the task is limited, and even more limited outside of English and other
mainstream languages, making building a reliable speech recognizer for many
languages even more challenging. One can even question if sequence-to-sequence
is a sustainable approach without fully utilizing data as well as the hybrid
systems.

On the other hand, the availability of unlabeled data is virtually unlimited,
and more importantly they can also be available under the form of pretrained
models. Since the release of Bidirectional Transformers or BERT [8] with
masked language model pretraining and its success in applying for natural
language processing, many Transformers [9] based architectures have been the
silver bullet to tackle low-resource problems. Recently, in the speech domain,
Transformers are also well adopted in supervised learning [2, 10] and
unsupervised pretraining with contrastive predicting methods [11, 12]. Most
importantly, these models can be effortlessly trained on a multilingual
dataset and acquire the acoustic or syntactic information of many languages.
Replacing the components in an e2e model with the pretrained ones is trivial
and can potentially gain largely for multilingual recognition.

With the pretrained models available in our arsenal, in this paper we set out
to explore the possibilities of combining them for multilingual ASR. While the
application wav2vec 2.0 and MBART50 individually is ubiqutuous and the
combination of them has been observed in Speech Translation [13], to the best
of our knowledge this is the first attempt in multilingual ASR, especially in
a large scale with many languages with different data size.

More importantly, there are many improvements for the Transformers over the
years that improve either modeling long range dependencies in self-attention
[14, 10] or adaptive components for multilingual models [15, 16]. It is
promising to also combine these techniques with the pretrained models for the
best results.

With such motivation, we carried out experiments with 32 languages ranging
from high to very low resource and explore the possibilities of using
pretrained models. Our contribution is as follows:

  * •

First, using multilingual pretrained acoustic and language models for the
encoder and decoder respective brings a large improvement as a whole. However
each pretrained module has a different influence on different languages,
namely encoder pretraining is more impactful for languages with higher
resources while the decoder counterpart is more effective for languages with
medium-low resources. Surprisingly, many languages with extremely low resource
(less than 5 hours) do not benefit much from this combination.

  * •

Second, the language specific modulation techniques such as language adapters
[15] and factorized adaptive weights [16] complement the two pretrained
modules very well and have a strong impact on especially the low-resource
languages mentioned above.

Moreover, there are different possibilities to integrate knowledge from
pretrained models, and it is not necessary by simply replacing components. We
also provided further analysis to the architecture, by showing that there are
benefits to either improving the self-attention mechanism by adding relative
positions during fine-tuning, or stacking the MBART encoders to the wav2vec
counterpart.

This is the continuation of the line of work building multilingual ASR systems
based on sequence-to-sequence neural networks [17, 5, 16]. Our work is
available for public at https://github.com/quanpn90/NMTGMinor providing a
highly CUDA-optimized implementation for both wav2vec and MBART which is
potentially useful for the community.

##  2 Modeling

Transformers [9] are a class of sequence-to-sequence models with an encoder
and a decoder with attention. Both components are equipped with self-attention
layers being able to well handle long-short range dependencies in the input,
output or alignment between sequences. These properties make this class of
model appeal for Speech Recognition [2, 3].

###  2.1 Pretrained Acoustic representation

Transformer encoders can be used to learn useful representation from input
masking and construction which is demonstrated in masked language model [8].
Following this trend, they are inevitably applied for audio signals, starting
from learning to reconstruct the log-mel frequency features [18] to using
quantization to learn latent variables [11].

In our work, the wav2vec 2.0 model [12] is selected to replace the typically
randomly initialized encoder. It consists of three main components: a
convolutional feature extractor that convolves and downsamples the raw audio
input, a deep Transformer encoder that learn high level representation from
the downsample sequence, and a quantization module to generate latent
variables. During pre-training, the network optimizes for a contrastive loss
function while masking the speech input. wav2vec 2.0 showed that it can
outperform other semi-supervised learning approaches using finetuning with a
CTC model.

###  2.2 Pretrained Multilingual BART

BART [19] was designed to learn syntactic features in languages using
Transformers. The network is tasked to reconstruct a particular sentence at
the decoder given a noisy version at the encoder side. MBART [20] and its
extension MBART50 [21] took the BART training scheme and applied to 50
languages. This proved to be very useful for multilingual translation by
finetuning the network on parallel data.

In speech recognition, data scarcity is even more problematic when the number
of sentences fall short compared to other natural language tasks. The presence
of MBART in the decoder is promising, especially for the self-attention parts
which play the role of a language model.

###  2.3 Language Adaptive Components

The development of multilingual models for either machine translation, speech
translation or speech recognition often concern between versatility versus
specialization [22]. The motivation comes from the disbelief that there are
features being shared between languages and at the same time each language
requires to selectively represented, and networks are encouraged to change
”modes” depending on the language being processed [23]. Since then, multingual
model designers opt to use specific network components being presented for
each language, ranging from weight generator [24] to adapters [15, 22] or
recently adaptive weights adding scales and biases to each weight matrix in
the whole architecture [16]. In this paper, the last two options are selected
for investigation thanks to being computationally manageable.

On the one hand, Adapters plugged into pretrained models were introduced in
computer vision [25] and later natural language processing [26] and recently
in Transformers for text/speech translation [15, 22]. They are materialized
with a small multilayer perceptrons (MLP) with one hidden layer that acts as a
downsampler (for parameter efficiency). This MLP is serialized at the end of
each layer in the Transformer to help the network changes the feature
distribution based on languages.

On the other hand, adaptive weights [16] was proposed based on the observation
that neural networks evolve rapidly yet the core remains to be matrix
multiplication. Therefore, it is possible to separate the weight matrix into a
shared component $W_{S}$ and language dependent adaptive scale $W_{ML}$ and
bias $W_{BL}$. The simple matrix multiplication $Y=WX$ becomes:

$Y=(W_{S}\cdot W_{ML}+W_{BL})^{\top}X$ (1)

In order to encourage the model to share parameters as well as keeping the
parameters efficient, the adaptive weights are factorized by using the form of
1-rank matrices [27] which can be compactly represented as a dot-product
between two vectors. This factorization can be established with $k$ vectors
per language so that there are $k$ independent weight factors followed by a
summation, which increases the rank of the additional weight matrices.

$\bar{W}=\sum_{i}^{k}r_{i}s_{i}^{\top}$ (2)

Equation 2 applies for all scale and bias matrices in the network.

Comparing two approaches, the advantage of the adapters is that they can
increase the depth of representation in the network thanks for having an
activation function in the downsampled layer. In contrast, the adaptive
weights have the advantage to directly affect each layer function, such as the
QKV-projection layer in self-attention, instead of applying a new function on
the output of the layer. In the Transformers specifically this particular
combination between a pretrained encoder and a decoder, the cross-attention
layers in the decoder are left with weights untrained to connect two
modalities audio and text. By being able to directly alter this function, the
advantage of the adaptive weights are even more considerable.

###  2.4 Related Work

Using pretrained models is very effective for low-resource machine translation
[20, 21] and recently speech translation [13] which can even handle zero
translation from speech. In ASR, the presence of pretrained acoustic models
[12, 28] allows recognition to be possible with very little data. The
combination of pretrained acoustic and language models are recently
investigated via learning to relax the modality mismatch [29, 30] yet still
requires CTC and limited in a monolingual setup. In terms of multilingual ASR,
there are various results in training a single model [5] that can overcome the
monolingual performance. In this paper, we combined most prominent techniques
to improve the results for many languages.

##  3 Experiments

There are 32 languages We report the error rates on the test sets of
CommonVoice and Europarl. It is notable that both of the pretrained model
(wav2vec) and the available supervised training data are mostly read speech,
leading to the curiosity about the performance on a more spontaneous setting.

Our speech recognition experiments are conducted using the public dataset
including CommonVoice [31] and Europarl [32] as training data.

For the progressive comparison, we trained a competitive supervised model
using the Transformer large configuration [9] with 24 encoder layers, 8
decoder layers and relative attention [10]. For transfer learning, we used the
wav2vec 2.0 model pretrained with $53$ languages [33] with the large
configuration that has the same hidden size with our initial model. It is
notable that the data used in pretraining is heavily biased to read speech
including CommonVoice and Multilingual Librispeech [34]. For pretrained
language models, MBART50 [21] with the same hidden size is used. For language
specific modules, we use adapters with hidden layer size $512$ and adaptive
weights with $k=8$ for bias and $k=1$ for scale matrices, so that they have
the same number of additional parameters per language. For training, we used
an effective batch size of around 2.84 hours of data per update, together with
a linear decay learning rate that peaks at $0.001$111The decaying equation
follows the same in Attention is all you need [9]. The supervised model takes
$150K$ updates to converge, while the model with transfer learning takes $50K$
updates.222Training was possible using 4 NVIDIA A100 GPUs.

The performance impact of the pretrained modules are fully presented in Table
1. The test data here is the combination of CommonVoice and Europarl wherein
the latter is available. Since the languages widely vary in terms of data
size, we divide them into three groups that have less than 10 hours (very
low), between 10-100 hours (low) and more than 100 hours (medium-large) of
training data. Notably, our supervised model outperformed the previously
reported error rates [16, 35].

Table 1: Performance(WER$\downarrow$) on the CommonVoice-Europarl dataset. Models include Transformers (TF), with wav2vec pre-training (W), with wav2vec and MBART50 (WM), with adapters (WMA) and factorized weights (WMF) and the version w/ frozen pt. weights. Language | Hours | TF | W | WM | WMA | WMF | FWMA | FWMF
---|---|---|---|---|---|---|---|---
(de) | 1050 | 10.4 | 8.1 | 7.8 | 7.7 | 7.2 | 11.8 | 10.0
(nl) | 150 | 13.2 | 8.4 | 7.7 | 7.4 | 6.8 | 10.2 | 9.0
(fr) | 800 | 15.2 | 12.7 | 12.12 | 11.62 | 11.2 | 16.7 | 15.1
(it) | 325 | 11.5 | 8.7 | 8 | 7.8 | 6.5 | 12.8 | 10.5
(fa) | 293 | 5.5 | 4.8 | 3.9 | 4.2 | 4.0 | 6.4 | 7.2
(pl) | 145 | 11.5 | 10.5 | 9.2 | 9.1 | 7.6 | 14.8 | 12.2
(pt) | 120 | 14.0 | 10.9 | 10.0 | 9.5 | 6.0 | 18.5 | 12.8
(es) | 400 | 10.9 | 8.0 | 7.6 | 7.4 | 6.2 | 11.2 | 9.8
(ru) | 148 | 10.0 | 8.7 | 5.5 | 5.7 | 5.4 | 20.1 | 10.1
(ta) | 198 | 28.6 | 24 | 20.2 | 20.7 | 21.0 | 31.4 | 31.5
(th) | 133 | 2.8 | 2.6 | 3.3 | 3.4 | 3.2 | 5.1 | 4.5
Average |  | 12.1 | 10.3 | 9.5 | 9.3 | 8.6 | 15.8 | 13
(ro) | 45 | 18.1 | 21.2 | 15.8 | 13.7 | 10.12 | 42.0 | 15.7
(ar) | 85 | 21.1 | 15.8 | 18.7 | 17.6 | 18.2 | 31.2 | 23.2
(et) | 32 | 30.4 | 22.1 | 14.8 | 15.1 | 13.2 | 32.5 | 21.5
(ja) | 26 | 13.0 | 10.3 | 8.3 | 7.91 | 7.9 | 21.8 | 11.5
(zh) | 63 | 25.9 | 16.7 | 15.2 | 14.6 | 14.7 | 37.6 | 18.2
(cs) | 49 | 19.8 | 15.8 | 10.0 | 9.2 | 8.4 | 16.4 | 12.7
(lt) | 16 | 43.3 | 37.9 | 31.9 | 26.7 | 25.5 | 71.3 | 30.0
(tr) | 30 | 10.4 | 13.6 | 7.5 | 8.4 | 7.5 | 9.8 | 10.1
(id) | 23 | 14.0 | 13.6 | 7.5 | 7.5 | 6.6 | 9 | 8.7
(mn) | 12 | 49.8 | 35.1 | 26.2 | 26.0 | 24.3 | 90.4 | 32.0
(sv) | 35 | 24.7 | 20.6 | 14.3 | 13.0 | 12.3 | 17.5 | 16.3
(uk) | 56 | 14 | 13.4 | 7.6 | 8.3 | 7.4 | 11.4 | 14.8
Average |  | 23.7 | 20 | 14.5 | 13.7 | 12.5 | 32.7 | 17.4
(lv) | 6 | 41.9 | 57.0 | 41.01 | 22.3 | 22.3 | 79.1 | 25.0
(vi) | 3 | 49.5 | 53.5 | 46.1 | 35.6 | 34.0 | 104.1 | 38.3
(ka) | 6 | 58.6 | 48.0 | 48.3 | 33.0 | 32.09 | 130.3 | 39.0
(sl) | 9 | 20.5 | 26.5 | 14.6 | 10.4 | 9.1 | 10.5 | 12.8
(fi) | 6 | 54.2 | 48.5 | 41.0 | 31.4 | 30.0 | 109.2 | 35.2
(hi) | 8 | 46.1 | 46.4 | 36.4 | 28.6 | 27.6 | 99.5 | 31.1
(gl) | 7 | 26.5 | 15.3 | 15.3 | 11.2 | 9.8 | 48.8 | 16.0
(ur) | 0.6 | 78.0 | 68.2 | 62.3 | 56.6 | 57.0 | 104.1 | 70.0
(kk) | 0.73 | 86.5 | 76.8 | 86.4 | 60.6 | 61.0 | 104.3 | 70.03
Average |  | 51.3 | 48.9 | 43.5 | 32.2 | 31.4 | 87.8 | 37.5
Overall |  | 30 | 24.8 | 20.9 | 17.4 | 16.6 | 41.6 | 21.6

###  3.1 Impact of acoustic pretraining

Compared to the Transformer model without any pretraining (TF), having the
encoder pretrained with XLS-R (W) brings a substantial improvement to the
average error rates by $18\%$, and this enables many languages to reach $10\%$
errors or lower. Across the three groups however, there is a clear difference
in impact. While the high-medium group enjoys a $20\%$ decrease in error rate,
this figure drops to $16\%$ in the first group, and the third group is only
improved by $4\%$.
While this is rather surprising, compared to the previously reported of
wav2vec 2.0 pretrained models on very low resource settings [12], it is
explainable by the difference of the approaches used in their and our works.
The pretrained acoustic model has often been used directly with the CTC loss
function to generate characters which heavily requires an external language
model. In our setting, we are limited in both acoustic and text resources for
the languages in the third group, and data scarcity makes learning to align
from attention [7] even harder.

###  3.2 Effects from pretrained language model

With that observation, initializing the decoder with the MBART pretrained
model is expected to alleviate the data scarcity problem. In fact, comparing
the model with two pretrained modules (WM) with the previous one showed a
benefit of $15\%$ error reduction for the former.

Some languages have worse performance with the MBART decoder, such as Arabic,
Turkish or Thai. This can be explained as a negative effect of the large byte-
pair encoding [36] model shared between many languages originally used with
MBART training. A large vocabulary size of $250K$ allow for large granularity
which is far from characters or phonemes, the units that speech recognition
models are often trained upon. Nevertheless, this is apparently not a problem
for most languages.

Analyzing the individual performance of each group, the medium-large group is
not impressively improved with just a $5\%$ reduction. This result shows that
having an additional six layers of decoders and even with pretrained weights
only has a minimal effect on the result and network depth has a clearer impact
on the source side than the target side in end-to-end speech recognition [2].
Probably the model does not struggle with the test data in terms of syntax,
despite the fact that the text data here is not comparable to typical language
models. Nevertheless, the second group receives a clearer merit from the
pretrained language model, by a $27\%$ improvement, and totally $39\%$
improvement compared to the original multilingual Transformer. Even with a
mismatched pretrained weights for the cross-attention module, it is still a
noticeable improvement coming from the pretrained self-attention, feed-forward
and layer normalization weights. The languages benefiting the most are
Romanian, Estonian, Czech, Turkish, Indonesian, Swedish and Ukrainian.

The effect on the third group is modest at $11\%$ reduction and the error
rates remain very high for Urdu (ur), Kazakh (kk), Finnish (fi), Latvian (lv)
or Vietnamese (vi). Many of the languages are also syntactically with many
morphological word forms, such as Finnish, making recognition even more
challenging. The most surprising improvement, however, comes from Slovenian
that massively reduces from $26\%$ to $14.6\%$. The overall struggling mainly
comes from data scarcity which is not adequate for the decoder cross-attention
layers.

###  3.3 Effect of the language-specific modules

As can be seen from Table 1, both techniques are able to help the model
generalize better in all language groups. Most importantly, the very low
resource group witnesses $27\%$ and $26\%$ improvement using adaptive weights
and adapters respectively, compared to the baseline model with two pretrained
modules.

In order to quantify their impact, we proceeded to freeze all of the
pretrained parameters and only fine-tuned the language-specific parameters.
There is a difference in how each technique handles this situation. With the
presence of only adapters, the errors in all languages deteriorate rapidly in
all language groups. Many languages in the low-group experience very bad
results including Romanian, Arabic, Chinese, Lithuanian and especially many
members within the very-low group exceed $90\%$ error rates. While this is
unexpected, we can see that the main problem here is the cross-attention layer
which is not familiar with the inputs coming from two modalities. The adapters
are not able to drive the bad context vectors (weighted-sum of the encoder
inputs) into meaningful representation in the low resource condition.

The adaptive factorized weights do not have this problem because they directly
alter cross-attention. As a result, the performance is much better than the
adapters even though they still fall behind the baseline without language-
specific modules.

###  3.4 Further analysis

In the previous sections, we presented the most important enhancement for our
multilingual setup coming from the pretrained modules and the language
adaptive components. The success of using the language adaptive components in
various places, either breaking the layer dynamics with adapters or the
function dynamics with the adaptive weights suggests that further improvement
can be found by adding more information to the system instead of treating the
pretrained model as an immovable black box.

Here we follow the implementation in [10] to add relative positions into the
wav2vec model, by adding the content-position interaction together with the
content bias and position bias to self-attention to all self-attention layers
of wav2vec with factorized weights. The additive information allowed for the
reduction of the average error rate to $16.04\%$ (3% improvement) and
especially an $12.7\%$ on the large-medium group. This evidently helps the
model learns better with adequate data.

Surprisingly, an alternative attempt to enhance the encoder simply by stacking
the MBART50 encoder on top of the wav2vec encoder (without any length
conversion) yields similar results, compared to the baseline WM, it improves
the large-medium group by an impressive $18.3\%$, yet only $3.2\%$ overall.
Not only does stacking the encoder increase the encoder depth, it also helps
the cross-attention layers because they are familiar with the output of the
text encoder during training. Training a stacked model with the adaptive
techniques would probably result in the best model in our experiments, however
it was unfortunately beyond our computational tolerance.

##  4 Conclusion

In this work, we massively improved multilingual ASR using a combination of
three techniques: encoder pretraining, decoder pretraining and adaptive
weights. Our empirical results indicate that each technique has a different
impact on different languages varying in linguistic characteristic and data
size. Nevertheless, at the very low condition the model still struggles to
reach an acceptable error rate. Our analysis in using adaptive weights shed
light on the future work, in which multimodal pretraining [37] is potentially
beneficial to address the cross-attention layers.

##  5 Acknowledgements

This work was realized within the project ELITR which has received funding
from the European Unions Horizon 2020 Research and Innovation Programme under
grant agreement No 825460.

## References

  * [1] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” _arXiv_ , 2019.
  * [2] N.-Q. Pham, T.-S. Nguyen, J. Niehues, M. Müller, and A. Waibel, “Very Deep Self-Attention Networks for End-to-End Speech Recognition,” in _Proc. Interspeech 2019_ , 2019, pp. 66–70.
  * [3] A. Gulati _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _arXiv preprint arXiv:2005.08100_ , 2020.
  * [4] T.-S. Nguyen, S. Stueker, J. Niehues, and A. Waibel, “Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation,” _arXiv preprint arXiv:1910.13296_ , 2019.
  * [5] V. Pratap, A. Sriram, P. Tomasello, A. Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert, “Massively multilingual asr: 50 languages, 1 model, 1 billion parameters,” _arXiv preprint arXiv:2007.03001_ , 2020.
  * [6] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016.
  * [7] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 4960–4964.
  * [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [10] N.-Q. Pham, T.-L. Ha, T.-N. Nguyen, T.-S. Nguyen, E. Salesky, S. Stüker, J. Niehues, and A. Waibel, “Relative Positional Encoding for Speech Recognition and Direct Translation,” in _Proc. Interspeech 2020_.
  * [11] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” _arXiv preprint arXiv:1904.05862_ , 2019.
  * [12] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [13] X. Li, C. Wang, Y. Tang, C. Tran, Y. Tang, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Multilingual speech translation with efficient finetuning of pretrained models,” _arXiv preprint arXiv:2010.12829_ , 2020.
  * [14] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * [15] A. Bapna, N. Arivazhagan, and O. Firat, “Simple, scalable adaptation for neural machine translation,” _arXiv preprint arXiv:1909.08478_ , 2019.
  * [16] N.-Q. Pham, T.-N. Nguyen, S. Stueker, and A. Waibel, “Efficient weight factorization for multilingual speech recognition,” _arXiv preprint arXiv:2105.03010_ , 2021.
  * [17] A. Kannan, A. Datta, T. N. Sainath, E. Weinstein, B. Ramabhadran, Y. Wu, A. Bapna, Z. Chen, and S. Lee, “Large-scale multilingual speech recognition with a streaming end-to-end model,” _arXiv preprint arXiv:1909.05330_ , 2019\.
  * [18] D. Jiang, X. Lei, W. Li, N. Luo, Y. Hu, W. Zou, and X. Li, “Improving transformer-based speech recognition using unsupervised pre-training,” _arXiv preprint arXiv:1910.09932_ , 2019.
  * [19] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” _arXiv preprint arXiv:1910.13461_ , 2019.
  * [20] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,” _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 726–742, 2020.
  * [21] Y. Tang, C. Tran, X. Li, P.-J. Chen, N. Goyal, V. Chaudhary, J. Gu, and A. Fan, “Multilingual translation with extensible multilingual pretraining and finetuning,” _arXiv preprint arXiv:2008.00401_ , 2020.
  * [22] H. Le, J. Pino, C. Wang, J. Gu, D. Schwab, and L. Besacier, “Lightweight adapter tuning for multilingual speech translation,” _arXiv preprint arXiv:2106.01463_ , 2021.
  * [23] J. B. Hampshire II and A. Waibel, “The meta-pi network: Building distributed knowledge representations for robust multisource pattern recognition,” _IEEE Computer Architecture Letters_ , 1992.
  * [24] E. A. Platanios, M. Sachan, G. Neubig, and T. Mitchell, “Contextual parameter generation for universal neural machine translation,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , Brussels, Belgium, 2018.
  * [25] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual domains with residual adapters,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [26] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2790–2799.
  * [27] Y. Wen, D. Tran, and J. Ba, “Batchensemble: an alternative approach to efficient ensemble and lifelong learning,” _arXiv preprint_ , 2020.
  * [28] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [29] G. Zheng, Y. Xiao, K. Gong, P. Zhou, X. Liang, and L. Lin, “Wav-bert: Cooperative acoustic and linguistic representation learning for low-resource speech recognition,” _arXiv preprint arXiv:2109.09161_ , 2021.
  * [30] C. Wang, Y. Wu, S. Liu, Z. Yang, and M. Zhou, “Bridging the gap between pre-training and fine-tuning for end-to-end speech translation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 9161–9168.
  * [31] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” _arXiv preprint arXiv:1912.06670_ , 2019.
  * [32] J. Iranzo-Sánchez, J. A. Silvestre-Cerdà, J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan, “Europarl-st: A multilingual corpus for speech translation of parliamentary debates,” in _ICASSP_ , 2020.
  * [33] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino _et al._ , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” _arXiv preprint arXiv:2111.09296_ , 2021.
  * [34] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” _arXiv preprint arXiv:2012.03411_ , 2020.
  * [35] W. Hou, H. Zhu, Y. Wang, J. Wang, T. Qin, R. Xu, and T. Shinozaki, “Exploiting adapters for cross-lingual low-resource speech recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 317–329, 2021.
  * [36] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2016.
  * [37] J. Ao, R. Wang, L. Zhou, S. Liu, S. Ren, Y. Wu, T. Ko, Q. Li, Y. Zhang, Z. Wei _et al._ , “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,” _arXiv preprint arXiv:2110.07205_ , 2021.


# Imagination-Augmented Natural Language Understanding

Yujie Lu1, Wanrong Zhu1, Xin Eric Wang2, Miguel Eckstein1, William Yang Wang1
1University of California, Santa Barbara, CA, USA
<EMAIL_ADDRESS>
2University of California, Santa Cruz, CA, USA
<EMAIL_ADDRESS>

###### Abstract

Human brains integrate linguistic and perceptual information simultaneously to
understand natural language, and hold the critical ability to render
imaginations. Such abilities enable us to construct new abstract concepts or
concrete objects, and are essential in involving practical knowledge to solve
problems in low-resource scenarios. However, most existing methods for Natural
Language Understanding (NLU) are mainly focused on textual signals. They do
not simulate human visual imagination ability, which hinders models from
inferring and learning efficiently from limited data samples. Therefore, we
introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural
language understanding tasks from a novel learning perspective—imagination-
augmented cross-modal understanding. iACE enables visual imagination with
external knowledge transferred from the powerful generative and pre-trained
vision-and-language models. Extensive experiments on GLUE Wang et al. (2018)
and SWAG Zellers et al. (2018) show that iACE achieves consistent improvement
over visually-supervised pre-trained models. More importantly, results in
extreme and normal few-shot settings validate the effectiveness of iACE in
low-resource natural language understanding circumstances.111Source code and
pre-trained models are publicly available at
https://github.com/YujieLu10/IACE-NLU

##  1 Introduction

Cognitive neuroscience studies reveal neural activation in vision-related
brain areas when reading text (Just et al., 2004) and show a tight
relationship between brain areas processing linguistic and visual semantic
information (Popham et al., 2021). In addition, visual imagery improves
comprehension during human language processing (Sadoski and Paivio, 1994).
Such imagination empowers human brains with generalization capability to solve
problems with limited supervision or data samples.

However, the field of Natural language Understanding has mainly been focused
on building machines based solely on language, ignoring the inherently
grounded imagination from the external visual world. These studies either
learn text-only representations from language corpora (Devlin et al., 2019;
Zhuang et al., 2021; Lan et al., 2020) or implicitly involve retrieved visual
supervision in pre-trained language models (Tan and Bansal, 2020). Thus, their
approaches appear limited in transferring the connection between language
understanding and visual imagination to downstream tasks, which are essential
to solving low-resource circumstances. In addition, these methods are limited
to text-only augmentations, whereas visual imaginations leverage cross-modal
augmentations to deal with low-resource situations.

Figure 1:  Rendering visual imagination is an intuitive way to activate
perception for linguistic understanding, e.g. natural language inference.

Human brains are multi-modal, integrating linguistic and perceptual
information simultaneously. Intuitively, the machines could achieve a higher-
level understanding of natural language and better learning transference by
imitating the procedure of human imagination behavior.

Inspired by this, we propose to understand language with the integration of
linguistic and perceptual information via introducing imagination supervision
into text-only NLU tasks. To imitate the imagination-augmented understanding
process as shown in Figure 1 with text-only data, we devise a procedure with
two steps: 1) pre-train a visually-supervised Transformer over paired text and
images retrieved from large-scale language corpus and image set, and 2)
construct the imagination with a generative model and fine-tune on downstream
NLU datasets by learning the paired imagination and natural language in a
cross-modal embedding. We show a detailed description of the cross-modal
imagination process for a specific Natural Language Inference task in Figure
2. In this way, we utilize machine imagination to improve the performance of
natural language understanding.

We adopt the few-shot learning setting to study the potential of using less
human effort of annotation for our proposed iACE to learn the natural language
with the help of imagination. Large margin performance gain in both extreme
and normal few-shot settings demonstrate the effectiveness of iACE in solving
problems with limited data samples. In the full data setting of GLUE Wang et
al. (2018) and SWAG Zellers et al. (2018), we observe the consistent
performance gain of our proposed iACE over the visually-supervised approach
(e.g., VOKEN Tan and Bansal (2020)) upon four language base models (e.g.,
BERT, RoBERTa).

In summary, the main contributions of our work are as follows:

  * •

We propose to solve the text-only learning problem in natural language
understanding tasks from a novel learning perspective: imagination-augmented
cross-modal language understanding.

  * •

To address the problem mentioned above, we devise iACE to generate
imaginations in a cross-modal representation space to guide the fine-tuning of
the visually supervised language models.

  * •

Experimental results in the few-shot setting validate the consistent
superiority of iACE over baselines in tackling the low-resource situation. In
full settings, iACE maintains the improvement in GLUE and SWAG.

Figure 2:  A detailed view of our iACE framework fine-tunes on natural
language inference task.

##  2 Related Work

#### Visually-aided Language Learning

Previous research attempt to introduce visual information to improve language
learning on various Natural Language Processing (NLP) scenarios, including but
not limited to machine translation (Grubinger et al., 2006; Elliott et al.,
2016), information retrieval (Funaki and Nakayama, 2015; Gu et al., 2018),
semantic parsing (Christie et al., 2016; Shi et al., 2019), natural language
inference (Xie et al., 2019), bilingual lexicon learning (Kiela et al., 2015;
Vulic et al., 2016), natural language generation evaluation (Zhu et al.,
2021), spatial commonsense reasoning (Liu et al., 2022) and language
representation learning (Lazaridou et al., 2015; Collell et al., 2017; Kiela
et al., 2018; Bordes et al., 2019; Lu et al., 2019; Li et al., 2019; Sun et
al., 2019; Luo et al., 2020; Chen et al., 2020; Li et al., 2020; Tan and
Bansal, 2020; Radford et al., 2021). While most of these studies acquire
visual information through retrieval from the web or large-scale image sets, a
recent line of studies attempt to generate visual supervision from scratch.
The visual information can either be provided in the form of representation
(Collell et al., 2017; Long et al., 2021) or concrete images (Gu et al., 2018;
Zhu et al., 2021). Though previous studies generate machine imagination, they
only tackle specific tasks, such as machine translation (Long et al., 2021) or
information retrieval (Gu et al., 2018). To the best of our knowledge, we are
the first to utilize machine abstract imagination from large pretrained vision
and language models to improve general NLU tasks. Recently, VOKEN Tan and
Bansal (2020) incorporate retrieved token-level visual information into
existing transformer models and achieve consistent improvement. iACE is
different from this work for two aspects: 1) we explicitly encode visual
imagination during fine-tuning. 2) we propose a novel model to borrow
knowledge from imagination in both training and inference.

#### Few-shot Natural Language Understanding

Natural Language Understanding (NLU) is a subfield in NLP that involves a
broad range of tasks such as question answering, sentiment analysis, and
textual entailment. Researchers have collected specific language corpus (Wang
et al., 2018; Zellers et al., 2018; Xu et al., 2020) to train the machines on
NLU learning. However, the general language understanding problem remains a
challenge. Few-shot learning is a learning paradigm that aims to predict the
correct class of instances with a relatively small amount of labeled training
examples (Fink, 2004; Fei-Fei et al., 2006). It has been receiving increasing
attention for its potential in reducing data collection effort and
computational costs and extending to rare cases. To deal with data-scarcity in
NLU problems, previous research introduces external knowledge (Sui et al.,
2021), utilizes meta-learning (Geng et al., 2019; Bansal et al., 2020; Han et
al., 2021) and adopts data augmentation to generate labeled utterances for
few-shot classes (Murty et al., 2021; Wei et al., 2021). Recent studies
(Radford et al., 2019; Brown et al., 2020) have shown that large-scale pre-
trained language models are able to perform NLU tasks in a few-shot learning
manner. The pre-trained multimodal models also display similar few-shot
learning ability (Tsimpoukelli et al., 2021). Unlike previous studies on pre-
trained multimodal Transformers that target solving multimodal tasks, our
study introduces imagination from the visual world into language models and
aims to improve NLU.

##  3 Our Approach

Figure 3:  Overview of iACE. The generator $\mathit{G}$ visualize imaginations
close to the encoded texts by minimizing $\mathcal{L}_{GAN}$. The cross-modal
encoder $\mathit{E}_{c}$ learns imagination-augmented language representation.
Two-step learning procedure consists of: 1) pre-train a Transformer with
visual supervision from large-scale language corpus and image set, 2) fine-
tune the visually supervised pre-trained Transformer and the imagination-
augmented cross-modal encoder on downstream tasks.

We illustrate how we solve the existing text-only learning problem in natural
language understanding tasks as the Imagination-augmented Cross-modal Language
Understanding (ICLU) problems in Section 3.1. Then we give a detailed
illustration of our proposed iACE’s architecture in Section 3.2. Finally, we
describe the procedure and training protocol of the perceptual-enhanced
linguistic understanding paradigm in Section 3.3.

###  3.1 Problem Definition

NLU is concerned with understanding the semantic meaning of the given
utterances. Data pieces for NLU can be structured as
$(x_{context},{\mathcal{X}},y)$, where $x_{context}$ represents the text
context, ${\mathcal{X}}=\\{x_{1},x_{2},...,x_{m},m\in\mathbb{N}\\}$ denote a
set of text snippets, and $m$ denotes the number of text samples for a
specific task. The model learns to predict the ground truth label $y$, which
is either regression or a classification label. While NLU is usually regarded
as a language-only task, we attempt to solve it from a cross-modal perspective
by introducing the novel ICLU problem.

In our ICLU problem, data pieces are structured as
$(x_{context},i_{context},{\mathcal{X}},{\mathcal{I}},y)$, in which
$i_{context}$ represents the visual context related to the text context, and
${\mathcal{I}}=\\{i_{1},i_{2},...,i_{n},n\in\mathbb{N}\\}$ denotes the
imagination set. The “imagination” refers to the images that are visualized
from the text. Here, $n$ is the number of visualized sentences for a specific
task, which is the same as $m$ by default.

To solve this problem, we devise a novel iACE to construct imagination from
textual data and learn the bi-directional alignment between the imagination
and text. Specifically, for each piece of text $x_{j}$ in the sentence set
${\mathcal{X}}$, we first follow (Esser et al., 2021; Radford et al., 2021)
and use a generative model to render a descriptive illustration $i_{j}$. The
visualized imagination will later serve as the visual input in the ICLU
problem.

###  3.2 Model Architecture

#### Overview

Figure 3 provides an overview of the iACE framework. iACE consists of two
modules: 1) the imagination generator $\mathit{G}$, 2) the imagination-
augmented cross-modal encoder $\mathit{E}_{c}$. Given the textual sentence
$x=\\{w_{1},w_{2},...,w_{k},k\in\mathbb{N}\\}$ ($w_{j}$ denotes the $j$-th
token in the sentence), $\mathit{G}$ generates corresponding visual
imagination $i$. The cross-modal encoder then encodes $x$ and $i$ as
${\bm{t}}$ and ${\bm{v}}$, respectively. iACE explicitly provides imagination
supervision to the visually-supervised Transformer during fine-tuning on
downstream NLU tasks.

#### Imagination Generator

Previous studies introduce visual supervision through retrieval from the web
or image sets. However, it is hard to find visuals that perfectly match the
topics discussed in each text snippet, especially for the relatively
complicated text input for the NLU tasks. Such misalignment between the input
text and the retrieved visuals might hinder the model from general language
understanding learning. Out of consideration for cross-modal feature
alignment, we choose to render specific visualization corresponding to each
piece of input text from scratch. Specifically, we construct imagination of
the textual input with a large-scale vision and language model guided
generative framework - VQGAN+CLIP (Crowson et al., 2022). For each piece of
input text $x$, we treat it as the prompt and use the VQGAN (Esser et al.,
2021) model to render the imagination $i$ with $128\times 128$ resolution and
$200$-step optimization. At each optimization step, we use the CLIP Radford et
al. (2021) model to assess how well the generated image corresponds to the
text.

$\mathcal{L}_{GAN}=2[\arcsin(\frac{1}{2}\lVert{\bm{t}}-{\bm{v}}\rVert)]^{2}$
(1)

To be specific, CLIP encodes the input text $x$ and the corresponding
imagination $i$ as ${\bm{t}}$ and ${\bm{v}}$, and the training objective is to
minimize the distance between ${\bm{t}}$ and ${\bm{v}}$ in the cross-modal
embedding space.

#### Cross-modal Encoder

We adopt CLIP as the cross-modal encoder to encode the input text and the
generated imaginations. CLIP Radford et al. (2021) is trained on large-scale
image-text pairs and is able to align visual and textual input in the
embedding space. Specifically, we use the $ViT-B/32$ version of Vision
Transformer as the image encoder, and Transformer Vaswani et al. (2017) with
the architecture modifications described in Radford et al. (2019) as the text
encoder. For each modality, the self-attention (SA) module is applied to model
the regions of imagination or the words of the text as follows:

$SA(F)=concat(softmax\frac{FW_{j}^{Q}{FW_{j}^{K}}^{\mathrm{T}}}{\sqrt{d_{k}}}FW_{j}^{V},...)W$
(2)

where $F$ denotes the set of regions of the imagination or the words of the
textual sentence. $W_{j}^{Q}$, $W_{j}^{K}$, and $W_{j}^{V}$ represents the
weight in the $j$-th head for query, key and value respectively. $d_{k}$ is
the dimension of the embedding. $W$ is the weight matrix for multiple heads.

To solve the ICLU problem, we learn the bi-directional relationship between
the text input and the visualized imagination. We apply late fusion on the
text feature ${\bm{t}}$ and visual feature ${\bm{v}}$ to construct the cross-
modal feature. Given the set of visual features $S_{\bm{v}}$ and textual
features $S_{\bm{t}}$, the fused embedding $X_{S}$ can be given with:

$X_{S}=[ReLU(W_{t}S_{\bm{t}}+b_{t}),ReLU(W_{j}S_{\bm{v}}+b_{j})]$ (3)

where $W$ and $b$ are of two separate fully connected layers to the visual and
text embeddings. The fused embeddings $X_{S}$ will go through two fully
connected layers before we receive the final imagination-augmented language
representation.

#### Visually-supervised Transformer

We implement the visually-supervised Transformer language model proposed in
Tan and Bansal (2020). The model architecture is a BERT-like pure-language-
based masked language model.

###  3.3 Learning Procedure

We introduce a novel paradigm to better understand natural language by
incorporating existing language models with visual imagination. As shown in
Figure 3, the procedure consists of two steps: (1) pre-train the visually-
supervised Transformer, and (2) fine-tune the framework with imagination on
downstream tasks.

#### Step 1: Visually-supervised Pre-training

We pre-train a visually-supervised Transformer following the scheme proposed
in VOKEN (Tan and Bansal, 2020), which extrapolates cross-modal alignments to
language-only data by contextually mapping language tokens to the related
images. In addition to masked language modeling, VOKEN proposed a voken
classification task: given a set of tokens with masks, the model is asked to
predict the best-matching image (the voken) for each tokens. The pre-training
loss can be given as:

$\mathcal{L}=-\lambda_{1}\sum_{w_{j}\in\hat{s}}\log
q_{j}(w_{j}|\check{s})-\lambda_{2}\sum_{w_{j}\in\hat{s}}\log
p_{j}(v(w_{j};s)|\check{s})$ (4)

Here $s$ is the token set, $\hat{s}$ is the masked tokens, and $\check{s}$ is
the unmasked tokens. The $q_{j}$ and $p_{j}$ represent the conditional
probability distribution of the $j$-th token given the token $w_{j}$ and voken
$v(w_{j};s)$ respectively, and $\lambda_{1}$ and $\lambda_{2}$ are the balance
factor of the masked language modeling task and the voken-classification task.
The cross-modal classification task enables the model to learn the matching
between the tokens from the language corpus (e.g., wiki) and its most-related
images from the image set (e.g., MSCOCO).

#### Step 2: Imagination-augmented Fine-tuning

We use GLUE (Wang et al., 2018) and SWAG (Zellers et al., 2018) as the
downstream datasets in the following sections. Our proposed iACE learns to
minimize the cross-entropy loss below:

$\mathcal{L}_{Imagine}=-\sum_{j=1}^{|D|}\sum_{k=1}^{K}y_{k}\log
p_{k}(d_{j}({\bm{t}};{\bm{v}})|D)$ (5)

where $j$ denotes the $j$-th data sample in dataset $D$, and $K$ os the class
number. The $p_{k}$ represents the conditional probability distribution of
$d_{j}$. During fine-tuning, the visually-supervised Transformer language
model only relied on the textual input to make predictions. The loss are
computed as:

$\mathcal{L}_{Lang}=-\sum_{j=1}^{|D|}\sum_{k=1}^{K}y_{k}\log
p_{k}(d_{j}({\bm{t}})|D)$ (6)

Notice that we use MSE loss for the regression task. The imagination-augmented
loss and pure-language based loss are summed up with a balance factor
$\lambda$ in a jointly training schema as:

$\mathcal{L}=\lambda\mathcal{L}_{Imagine}+(1-\lambda)\mathcal{L}_{Lang}$ (7)

We use Adam Optimizer with a learning rate $1e-4$ for the GLUE benchmark and
$2e-5$ for the SWAG dataset. We discuss more details in Section 4.

##  4 Experiments

| SST-2 | QNLI | QQP | MNLI
---|---|---|---|---
Extreme Few-shot | $0.1\%$ | $0.3\%$ | $0.5\%$ | $0.1\%$ | $0.3\%$ | $0.5\%$ | $0.1\%$ | $0.3\%$ | $0.5\%$ | $0.1\%$ | $0.3\%$ | $0.5\%$
$VOKEN(Bert_{base})$ | 54.70 | 77.98 | 80.73 | 50.54 | 51.60 | 61.96 | 44.10 | 60.65 | 65.46 | 37.31 | 54.62 | 58.79
$iACE(Bert_{base})$ | 77.98 | 80.96 | 81.42 | 51.64 | 58.33 | 64.03 | 49.36 | 63.67 | 71.17 | 40.07 | 56.49 | 59.57
$VOKEN(Roberta_{base})$ | 70.99 | 71.10 | 77.86 | 54.37 | 62.23 | 65.78 | 62.32 | 67.25 | 70.18 | 48.59 | 49.76 | 58.23
$iACE(Roberta_{base})$ | 75.34 | 78.66 | 83.60 | 54.79 | 65.03 | 65.83 | 65.43 | 68.11 | 70.77 | 48.94 | 52.74 | 59.39
Normal Few-shot | $1\%$ | $3\%$ | $5\%$ | $1\%$ | $3\%$ | $5\%$ | $1\%$ | $3\%$ | $5\%$ | $1\%$ | $3\%$ | $5\%$
$VOKEN(Bert_{base})$ | 81.40 | 86.01 | 84.75 | 64.17 | 77.36 | 80.19 | 72.55 | 78.37 | 80.50 | 60.45 | 62.73 | 72.35
$iACE(Bert_{base})$ | 82.45 | 87.04 | 86.47 | 65.09 | 79.54 | 80.52 | 74.31 | 78.69 | 80.52 | 62.15 | 70.43 | 73.73
$VOKEN(Roberta_{base})$ | 83.78 | 84.08 | 87.61 | 75.00 | 81.16 | 81.23 | 73.14 | 79.09 | 79.63 | 63.51 | 70.68 | 74.02
$iACE(Roberta_{base})$ | 83.83 | 84.63 | 89.11 | 79.35 | 81.41 | 81.65 | 73.72 | 79.38 | 79.81 | 65.66 | 70.76 | 74.10

Table 1: Model-agnostic Improvement in Few-shot Setting. iACE and VOKEN upon
BERT and RoBERTa base size architecture are fine-tuned in Extreme Few-shot
($0.1\%$, $0.3\%$, $0.5\%$) and Normal Few-shot setting ($1\%$, $3\%$, $5\%$).
For the few-shot setting, we use large and stable datasets from GLUE
Benchmark. We compare accuracy on SST-2, QNLI, QQP, and MNLI and the average
of accuracy and F1 score on QQP. BEST results are highlighted.

###  4.1 Experimental Setup

#### Datasets & Metric

We conduct experiments to evaluate the performance of our proposed method over
SST-2 (Socher et al., 2013), QNLI (Rajpurkar et al., 2016), QQP (Iyer et al.,
2017), MultiNLI Williams et al. (2018), MRPC (Dolan and Brockett, 2005), STS-B
(Agirre et al., 2007) from GLUE Wang et al. (2018) Benchmark, and SWAG Zellers
et al. (2018) dataset. We construct few-shot setting subsets by taking
$0.1\%$, $0.3\%$, and $0.5\%$ of training instances as the Extreme Few-shot
Setting, and $1\%$, $3\%$, and $5\%$ as the Normal Few-shot Setting. We train
the model with the subsets and evaluate its performance on the complete
development set. We use accuracy as the default evaluation metric and compare
such results in the following sections.

#### Baselines

We choose BERT Devlin et al. (2019) and RoBERTa Zhuang et al. (2021) as the
base language models, and apply our iACE framework on top of their small and
base architectures for comparison. A recent study proposes a visually-
supervised language model VOKEN Tan and Bansal (2020) that introduces visual
supervision into language model pre-training by borrowing external knowledge
from retrieved images of the tokens. In natural language understanding tasks,
VOKEN achieved improvements over language-based baselines BERT and RoBERTa.
Thus we also use VOKEN built upon these language-based models as a set of
powerful baselines. In the following experiments, each model is first pre-
trained with visual supervision introduced in Tan and Bansal (2020) upon the
four base models (BERTsmall, BERTbase, RoBERTasmall and RoBERTabase). Then the
models will be fine-tuned on downstream tasks.

Notice that base models and VOKEN use pure-language training objectives during
fine-tuning. Neither of them utilizes the visual signals inherent in the
downstream language corpora. In contrast, our iACE explicitly introduces
visual imagination supervisions into fine-tuning and inference stages.

#### Implementation Details

We train RoBERTa with the same configurations as a robustly optimized pre-
training approach based on BERT of the same size. BERTsmall has $6$ repeating
layers, $512$ hidden dimension. BERTbase has $12$ repeating layers, $768$
hidden dimension.

The imagination of the texts is generated interactively by using VQGAN+CLIP,
with $128\times 128$ size, $500$ iterations. We use pre-trained VQGAN
(imagenetf16) and CLIP (ViT-B/$32$). We leverage CLIP (ViT-B/$32$) as our
language and vision model for premise and hypothesis, and imagination of them.
The text and image dimension is $512$. The dropout rate is set to $0.1$. We
use Cross-Entropy loss for our cross-modal classification. Each model was
first pre-trained on $4$ TITAN RX GPUs for $30$ epochs with early stopping and
a batch size of $32$ and a sequence length of $126$. The optimizer used is
Adam with a learning rate of $2e-4$ and a weight decay of $0.01$. The models
are then fine-tuned on GLUE benchmark and SWAG dataset for $3$ epochs with
$32$ batch size. We adopt the joint training strategy for our proposed iACE
and visually supervised transformer during fine-tuning. The learning rate of
the Adam optimizer is set as $1e-4$ and $2e-5$ for GLUE and SWAG,
respectively.

|  | SST-2 | QNLI | QQP | MNLI | ALL
---|---|---|---|---|---|---
Base Model | Method | $0.1\%$ | $1.0\%$ | $3.0\%$ | $0.1\%$ | $1.0\%$ | $3.0\%$ | $0.1\%$ | $1.0\%$ | $3.0\%$ | $0.1\%$ | $1.0\%$ | $3.0\%$ | Avg.
BERTbase | Direction | 49.01 | 79.59 | 87.15 | 51.31 | 52.55 | 66.90 | 56.74 | 31.58 | 31.59 | 32.73 | 61.54 | 70.72 | 55.95
BERTbase | Unify | 48.96 | 77.98 | 86.92 | 50.54 | 52.02 | 67.20 | 55.29 | 56.93 | 79.09 | 39.05 | 63.29 | 70.86 | 62.34
BERTbase | iACE | 77.98 | 82.45 | 87.04 | 51.64 | 65.09 | 79.54 | 49.36 | 74.31 | 78.69 | 40.07 | 62.15 | 70.43 | 68.23
RoBERTabase | Direction | 72.71 | 80.38 | 84.63 | 54.91 | 74.68 | 78.58 | 61.57 | 74.68 | 31.59 | 32.95 | 61.96 | 70.62 | 64.94
RoBERTabase | Unify | 75.11 | 80.04 | 88.07 | 53.62 | 74.64 | 78.47 | 64.94 | 74.85 | 76.84 | 51.12 | 65.42 | 70.74 | 71.15
RoBERTabase | iACE | 75.34 | 83.83 | 84.63 | 54.79 | 79.35 | 81.41 | 65.43 | 73.72 | 79.38 | 48.94 | 65.66 | 70.76 | 71.93

Table 2: Method Design Ablation in Few-shot Setting. We compare the results of
two variants over $0.1\%$, $1.0\%$, $3.0\%$ of SST-2, QNLI, QQP and MNLI
dataset. Details of Direction and Unify are illustrated in Section 4.3.

|  | Extreme Few-shot (0.1%) | Normal Few-shot (3.0%) | ALL
---|---|---|---|---
Base Model | Composition | SST-2 | QNLI | QQP | MNLI | SST-2 | QNLI | QQP | MNLI | Avg.
BERTbase | Textual-Only | 49.08 | 50.54 | 55.48 | 38.82 | 87.50 | 67.05 | 77.42 | 71.00 | 62.11
BERTbase | Visual-Only | 59.97 | 50.56 | 49.01 | 39.05 | 86.81 | 67.23 | 79.06 | 70.80 | 62.81
BERTbase | Visual+Textual (VT) | 53.89 | 50.54 | 49.15 | 38.83 | 87.04 | 66.81 | 79.16 | 70.77 | 62.02
BERTbase | Bi-directional VT | 77.98 | 51.64 | 49.36 | 40.07 | 87.04 | 79.54 | 78.69 | 70.43 | 66.84
RoBERTabase | Textual-Only | 75.57 | 53.85 | 64.96 | 35.28 | 84.07 | 78.51 | 75.76 | 51.48 | 64.93
RoBERTabase | Visual-Only | 75.11 | 54.18 | 65.01 | 47.22 | 84.17 | 79.88 | 76.88 | 70.56 | 69.12
RoBERTabase | Visual+Textual (VT) | 74.20 | 53.98 | 65.43 | 47.35 | 83.94 | 79.96 | 76.87 | 70.73 | 69.05
RoBERTabase | Bi-directional VT | 75.34 | 54.79 | 65.43 | 48.94 | 84.63 | 81.41 | 79.38 | 70.76 | 70.08

Table 3: Imagination Composition Ablation in Few-shot Setting. Bi-directional
VT represents the full input for iACE. More details about Textual Only, Visual
Only and Visual+Textual are illustrated in Section 4.3.

###  4.2 Few-shot Learning Results

We claim that introducing imagination into language processing helps the
existing language-based system tackle the low-resource situation. Thus, the
automatically generated imagination helps reduce the human effort to annotate
textual data. To verify this, we define two situations, a normal few-shot
setting, and an extreme few-shot setting. For the normal few-shot setting, we
keep $1\%$, $3\%$, and $5\%$ of the training dataset for each task in GLUE
Benchmark. For the extreme few-shot setting, we keep a lower number of the
training dataset, which is reduced to $0.1\%$, $0.3\%$, and $0.5\%$ of the
training dataset. We train the models with the same configuration under these
two settings and compare them with visually supervised transformer baselines
to confirm the benefit that our proposed iACE brings to the few-shot
situation.

Results of the few-shot setting are reported in Table 1. Following Tan and
Bansal (2020), we only report the four largest and stable tasks in GLUE for
the model-agnostic comparison. We report the accuracy for SST-2, QNLI, MNLI.
For QQP and MRPC, we report the average of F1 and accuracy. For SWAG, we
report the correlation. We observe that the imagination information remarkably
helps with both the normal few-shot curriculum and extreme few-shot
curriculum. We assume the imagination-augmented fine-tuning successfully
transfers the language understanding from the large-scale vision and language
model. Thus iACE achieves consistent performance gain and shows great
superiority of generalization and transferring ability.

Base Model | Method | SST-2 | QNLI | QQP | MNLI | MRPC | STS-B | SWAG | Avg.
---|---|---|---|---|---|---|---|---|---
BERTsmall | VOKEN | 89.7 | 85.0 | 87.3 | 78.6 | 78.2 | 80.4 | 57.6 | 79.5
BERTsmall | iACE | 89.8 | 86.2 | 87.7 | 78.9 | 78.4 | 82.7 | 57.9 | 80.2
BERTbase | VOKEN | 92.2 | 88.6 | 88.6 | 82.6 | 83.5 | 86.0 | 70.6 | 84.6
BERTbase | iACE | 91.7 | 88.6 | 89.1 | 82.8 | 85.8 | 86.6 | 70.8 | 85.1
RoBERTasmall | VOKEN | 87.8 | 85.1 | 85.3 | 76.5 | 78.5 | 78.6 | 53.6 | 77.9
RoBERTasmall | iACE | 89.2 | 85.1 | 86.5 | 76.8 | 79.0 | 78.7 | 53.7 | 78.3
RoBERTabase | VOKEN | 90.5 | 89.2 | 87.8 | 81.0 | 87.0 | 86.9 | 68.5 | 84.4
RoBERTabase | iACE | 91.6 | 89.1 | 87.9 | 82.6 | 87.7 | 86.9 | 68.5 | 84.9
Table 4: Model-agnostic Improvement in Full Data Setting. Results of iACE and
VOKEN upon BERT and RoBERTa of small($6L/512H$) and base($12L/768H$)
architecture are reported. The models are fine-tuned over GLUE Benchmark and
SWAG with access to the full dataset. BEST results are highlighted.

###  4.3 Ablation Studies

We conduct ablation studies over both the method side and data side to
validate their contribution to our proposed iACE.

#### Method Design Ablation

Two method variants of our imagination-augmented encoder are built as
baselines to validate the importance of our bi-directional cross-modal
imagination design in iACE. The variants are built upon RoBERTabase and
BERTbase base models. Specifically, we develop variant Direction and Unify.
Direction represent alignment between text input and imagination into a
directional embedding as FUSE(${\bm{t}}_{sen1}-{\bm{i}}_{sen1}$,
${\bm{t}}_{sen2}-{\bm{i}}_{sen2}$). Unify encode the text and imagination,
considering the direction from vision to language by encoding as
FUSE(${\bm{t}}_{sent1}$, ${\bm{t}}_{sent2}$, ${\bm{i}}_{sent1}$,
${\bm{i}}_{sent2}$). While iACE consider direction from visoin to language and
language to vision by encoding as the combination of FUSE(${\bm{t}}_{sent1}$,
${\bm{i}}_{sent2}$) and FUSE(${\bm{i}}_{sent1}$, ${\bm{t}}_{sent2}$). As shown
in Table 2, our bi-directional imagination and language learning achieve
stable and best average performance. These results indicate that our bi-
directional imagination method design obtain generalization and transferring
ability. We assume iACE benefits from both learning from language to vision
and learning from vision to language simultaneously.

#### Imagination Composition Ablation

The composition of the imagination is essential for the performance. To
further study the importance of full imagination, we ablate the data side by
constructing a textual-only model denoted as Textual Only, a visual-only
imagination denoted as Visual Only and a single directional imagination input
denoted as Visual+Textual. Visual Only and Visual+Textual represent the
imagination model use visual pairs $({\bm{i}}_{sent1},{\bm{i}}_{sent2})$ and
one direction visual and textual pairs $({\bm{i}}_{sent1},{\bm{t}}_{sent2})$
as input respectively. Our full approach use Bi-directional VT which takes
$({\bm{i}}_{sent1},{\bm{t}}_{sent2})$ and
$({\bm{t}}_{sent1},{\bm{i}}_{sent2})$ as input.

Results are reported in Table 3 for Extreme Few-shot setting and normal few-
shot setting. We observe Bi-directional VT data input achieve the most stable
and the best average performance. Results show the importance of bi-
directional imagination from all the textual input to construct an
imagination-augmented cross-modal encoder.

Figure 4: Case studies on the STS-B and SNLI tasks. The baseline models yield
predictions solely based on the text input, while our approach takes both the
text input and corresponding visualization into consideration. On both tasks,
our iACE gives predictions that are more aligned with the ground truth.

###  4.4 Model-agnostic Improvement

iACE is a model-agnostic training paradigm that could help existing models
achieve consistent gain over GLUE and SWAG with both the few-shot setting and
full data setting. To validate such model-agnostic effectiveness of our
proposed novel paradigm in processing natural language, we compare the
performance with two language models (BERT and RoBERTa) of two architectures
("6L/512H" and "12L/768H"), and a strong visually supervised pre-trained
baseline VOKEN Tan and Bansal (2020).

Table 4 shows the metric comparison on GLUE and SWAG. The base models are
trained with a masked language model. The VOKEN model is pre-trained with a
masked language model with an additional voken-classification task as
introduced visual supervision. iACE achieves model-agnostic improvement over
the model that solely fine-tune based on textual information, including the
pure-language-based model and visually supervised pre-trained model. The gain
is consistently observed from different architectures of models.

###  4.5 Case Study

Figure 4 lists out our examples for the case study. We show the results from
the natural language inference and sentence similarity task. We use examples
from the STS-B and SNLI datasets. Our contextual imagination describes the
textual input as expected and provides an external prediction reference.

For example (a), given the structurally diversified sentence and low $n$-grams
overlaps but high semantic similarity, we observe that the pure language-based
model predicts the wrong label. While the imagination helps the model capture
the semantic similarity between two textual inputs via comparing the cross-
modal semantics with the imagination information. From example (b), we observe
the pure language-based model predicts the wrong label based on the similar
sentence structure and high $n$-grams overlaps. While the imagination helps
the model capture the difference between the similar premise and hypothesis
text.

##  5 Conclusion

We treat the text-only learning problem in Natural Language Understanding
tasks as a cross-modal language understanding problem with generated
imagination as supervision. In this scenario, the task aims to bridge the gap
between the human and the agent language understanding in both linguistic and
perceptual procedures. To address the proposed problem, we devised a model-
agnostic learning paradigm iACE. Specifically, we build the imagination of the
downstream dataset using an interactive generative approach with guidance from
a self-supervised pre-trained large-scale image and text model. Our proposed
iACE surpassed baselines of two architecture sizes by a large margin in the
few-shot setting. The improvement is consistently observed over pure-language
baselines (BERT and RoBERTa) and visually supervised VOKEN on the GLUE and
SWAG dataset. The results show the superiority of our iACE in language
understanding and handling low-resource circumstances.

## Acknowledgement

We would like to thank the Robert N. Noyce Trust for their generous gift to
the University of California via the Noyce Initiative. The research was also
sponsored by the U.S. Army Research Office and was accomplished under Contract
Number W911NF19-D-0001 for the Institute for Collaborative Biotechnologies.
The writers’ opinions and conclusions in this publication are their own and
should not be construed as representing the sponsors or the U.S. government’s
official policy, expressed or inferred. Regardless of any copyright notation
herein, the United States government is authorized to reproduce and distribute
reprints for government purposes.

## Ethical Statement

In this study, we only cover NLU datasets with English annotations. Such
limitation is since the large-scale pre-trained multimodal models used in our
studies, such as CLIP and VQGAN, are only trained on English corpus as of the
date we conduct the experiments 222As of Dec. 2021..

This study use CLIP and VQGAN to render images given the text prompt. Suppose
there exists any bias in the training dataset for the large-scale pre-trained
multimodal models used in our study. In that case, our “imagination” approach
may face an issue of fairness since the visual generative model might be more
likely to illustrate specific types of images that it has seen in the training
data. Moreover, if the training dataset for CLIP or VQGAN contains any
personal information, then our “imagination” approach may strike a threat on
privacy leakage given certain triggers or prompts. Even though we did not
witness such issues in our study, we should keep in mind that the
aforementioned behaviors would impair iACE’s effectiveness.

## References

  * Agirre et al. (2007) Eneko Agirre, Llu’is M‘arquez, and Richard Wicentowski. 2007.  Semantic textual similarity benchmark.  In _Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)_ , Prague, Czech Republic. Computational Linguistics.
  * Bansal et al. (2020) Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020.  Learning to few-shot learn across diverse natural language classification tasks.  In _COLING_.
  * Bordes et al. (2019) Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. 2019.  Incorporating visual semantics into sentence representations within a grounded space.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 696–707, Hong Kong, China. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _ECCV_.
  * Christie et al. (2016) Gordon Christie, Ankit Laddha, Aishwarya Agrawal, Stanislaw Antol, Yash Goyal, Kevin Kochersberger, and Dhruv Batra. 2016.  Resolving language and vision ambiguities together: Joint segmentation & prepositional attachment resolution in captioned scenes.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1493–1503, Austin, Texas. Association for Computational Linguistics.
  * Collell et al. (2017) Guillem Collell, Ted Zhang, and Marie-Francine Moens. 2017.  Imagined visual representations as multimodal embeddings.  In _AAAI_.
  * Crowson et al. (2022) Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. 2022.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  _arXiv preprint arXiv:2204.08583_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30k: Multilingual english-german image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language, hosted by the 54th Annual Meeting of the Association for Computational Linguistics, VL@ACL 2016, August 12, Berlin, Germany_ , pages 70–74. Association for Computational Linguistics (ACL).  5th Workshop on Vision and Language, VL 2016 ; Conference date: 12-08-2016 Through 12-08-2016.
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12873–12883.
  * Fei-Fei et al. (2006) Li Fei-Fei, Rob Fergus, and Pietro Perona. 2006.  One-shot learning of object categories.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 28:594–611.
  * Fink (2004) Michael Fink. 2004.  Object classification from a single example utilizing class relevance metrics.  In _NIPS_.
  * Funaki and Nakayama (2015) Ruka Funaki and Hideki Nakayama. 2015.  Image-mediated learning for zero-shot cross-lingual document retrieval.  In _EMNLP_.
  * Geng et al. (2019) Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019\.  Induction networks for few-shot text classification.  In _EMNLP_.
  * Grubinger et al. (2006) Michael Grubinger, Paul D. Clough, Henning Müller, and Thomas Deselaers. 2006\.  The iapr tc-12 benchmark: A new evaluation resource for visual information systems.
  * Gu et al. (2018) Jiuxiang Gu, Jianfei Cai, Shafiq R. Joty, Li Niu, and G. Wang. 2018.  Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7181–7189.
  * Han et al. (2021) Chengcheng Han, Zeqiu Fan, Dongxiang Zhang, Minghui Qiu, Ming Gao, and Aoying Zhou. 2021.  Meta-learning adversarial domain adaptation network for few-shot text classification.  In _FINDINGS_.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017.  First quora dataset release: Question pairs.
  * Just et al. (2004) M. Just, S. Newman, T. Keller, A. McEleney, and P. Carpenter. 2004.  Imagery in sentence comprehension: an fmri study.  _NeuroImage_ , 21:112–124.
  * Kiela et al. (2018) Douwe Kiela, Alexis Conneau, A. Jabri, and Maximilian Nickel. 2018.  Learning visually grounded sentence representations.  In _NAACL_.
  * Kiela et al. (2015) Douwe Kiela, Ivan Vulic, and Stephen Clark. 2015.  Visual bilingual lexicon induction with transferred convnet features.  In _EMNLP_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.
  * Lazaridou et al. (2015) Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015.  Combining language and vision with a multimodal skip-gram model.  In _NAACL_.
  * Li et al. (2020) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 11336–11344. AAAI Press.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _ArXiv_ , abs/1908.03557.
  * Liu et al. (2022) Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022.  Things not written in text: Exploring spatial commonsense from visual signals.
  * Long et al. (2021) Quanyu Long, Mingxuan Wang, and Lei Li. 2021.  Generative imagination elevates machine translation.  In _NAACL_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _NeurIPS_.
  * Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. 2020.  Univilm: A unified video and language pre-training model for multimodal understanding and generation.  _ArXiv_ , abs/2002.06353.
  * Murty et al. (2021) Shikhar Murty, Tatsunori B. Hashimoto, and Christopher D. Manning. 2021.  Dreca: A general task augmentation strategy for few-shot natural language inference.  In _NAACL_.
  * Popham et al. (2021) Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James S Gao, Anwar O Nunez-Elizalde, and Jack L Gallant. 2021.  Visual and linguistic semantic representations are aligned at the border of human visual cortex.  _Nature neuroscience_ , 24(11):1628—1636.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Sadoski and Paivio (1994) Mark Sadoski and A. Paivio. 1994.  A dual coding view of imagery and verbal processes in reading comprehension.
  * Shi et al. (2019) Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019.  Visually grounded neural syntax acquisition.  In _ACL_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _EMNLP_.
  * Sui et al. (2021) Dianbo Sui, Yubo Chen, Binjie Mao, Delai Qiu, Kang Liu, and Jun Zhao. 2021.  Knowledge guided metric learning for few-shot text classification.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3266–3271, Online. Association for Computational Linguistics.
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin P. Murphy, and Cordelia Schmid. 2019\.  Videobert: A joint model for video and language representation learning.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7463–7472.
  * Tan and Bansal (2020) Haochen Tan and Mohit Bansal. 2020.  Vokenization: Improving language understanding via contextualized, visually-grounded supervision.  In _EMNLP_.
  * Tsimpoukelli et al. (2021) Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021.  Multimodal few-shot learning with frozen language models.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 200–212. Curran Associates, Inc.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vulic et al. (2016) Ivan Vulic, Douwe Kiela, Stephen Clark, and Marie-Francine Moens. 2016.  Multi-modal representations for improved bilingual lexicon learning.  In _ACL_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wei et al. (2021) Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, and Shiqi Xu. 2021.  Few-shot text classification with triplet networks, data augmentation, and curriculum learning.  In _NAACL_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Xie et al. (2019) Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019.  Visual entailment: A novel task for fine-grained image understanding.  _ArXiv_ , abs/1901.06706.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020.  CLUE: A Chinese language understanding evaluation benchmark.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  In _EMNLP_.
  * Zhu et al. (2021) Wanrong Zhu, Xin Eric Wang, An Yan, Miguel P. Eckstein, and William Yang Wang. 2021\.  Imagine: An imagination-based automatic evaluation metric for natural language generation.  _ArXiv_ , abs/2106.05970.
  * Zhuang et al. (2021) Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021.  A robustly optimized BERT pre-training approach with post-training.  In _Proceedings of the 20th Chinese National Conference on Computational Linguistics_ , pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China.