#  Can Language Models Understand Physical Concepts?

Lei Li1, Jingjing Xu2, Qingxiu Dong1, Ce Zheng1, Qi Liu3, Lingpeng Kong3, Xu
Sun1
1National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
2Shanghai AI Lab, 3The University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>{liuqi<EMAIL_ADDRESS>
###### Abstract

Language models (LMs) gradually become general-purpose interfaces in the
interactive and embodied world, where the understanding of physical concepts
is an essential prerequisite. However, it is not yet clear whether LMs can
understand physical concepts in the human world. To investigate this, we
design a benchmark VEC that covers the tasks of (i) Visual concepts, such as
the shape and material of objects, and (ii) Embodied Concepts, learned from
the interaction with the world such as the temperature of objects. Our zero
(few)-shot prompting results show that the understanding of certain visual
concepts emerges as scaling up LMs, but there are still basic concepts to
which the scaling law does not apply. For example, OPT-175B performs close to
humans with a zero-shot accuracy of $85$% on the material concept, yet behaves
like random guessing on the mass concept. Instead, vision-augmented LMs such
as CLIP and BLIP achieve a human-level understanding of embodied concepts.
Analysis indicates that the rich semantics in visual representation can serve
as a valuable source of embodied knowledge. Inspired by this, we propose a
distillation method to transfer embodied knowledge from VLMs to LMs, achieving
performance gain comparable with that by scaling up parameters of LMs
$134\times$. 111Our dataset is available at https://github.com/TobiasLee/VEC

##  1 Introduction

With the emergent capabilities such as arithmetic (Brown et al., 2020; Wei et
al., 2022) and multi-step reasoning (Chowdhery et al., 2022) brought by large-
scale pre-training, language models (LMs) are gradually becoming unified
interfaces (Hao et al., 2022), capable of instructing embodied robots for
high-level tasks such as _cleaning the spilled coke_ in interactive and
embodied environments (Ahn et al., 2022). Understanding physical concepts is
an essential prerequisite for these tasks, e.g., producing correct
instructions for cleaning the coke requires understanding the visual
characteristics of a coke can, as well as physical properties such as
hardness. However, it still remains unclear whether current LMs can understand
basic physical concepts (Driess et al., 2023).

Concpet Category | Instance | Label | # of Examples
---|---|---|---
| Color |  $h$: melon, $t_{1}$: green, $t_{2}$: black | green | 574
| Shape |  $h$: lemon, $t_{1}$: triangle, $t_{2}$: round | round | 140
Visual Concepts | Material |  $h$: guitar, $t_{1}$: wood, $t_{2}$: glass | wood | 284
Size |  $h$: ant, $r$: larger than, $t$: bird | false | 500
Height |  $h$: bottle, $r$: shorter than, $t$: truck | true | 500
Mass |  $h$: wooden spoon, $r$: heavier than, $t$: toaster | false | 654
Embodied Concepts | Temperature |  $h$: ice, $r$: colder than, $t$: water | true | 422
Hardness |  $h$: pearl, $r$: softer than, $t$: glass | true | 1,016

Table 1: The illustration of VEC benchmark. We design two forms of probing
tasks. The former (Color, Shape and Material) asks models to make a choice
between two tail options given the head object. The latter (Size, Height, and
all embodied concepts) requires LMs to judge whether the relation is valid
given the head and the tail.

To answer the question, we first define an evaluation suite of physical
concepts covering visual and embodied concepts. Specifically, _visual
concepts_ examine knowledge that can be gained via visual perception,
including generic visual concepts, such as color, shape, and material of
common objects, and spatial perception, which focuses on the relationship
between visual stimuli, i.e., relative size and height of objects. The ability
to deal with visual concepts serves as the basis for understanding real-world
scenes to perform further instruction. _Embodied concepts_ examine knowledge
that requires more interaction and multimodal sensory experience in the
embodied world, including knowledge about the mass, temperature, and hardness
of objects, e.g., ice is colder than water. Understanding embodied concepts is
essential for an embodied agent to make correct choices when translating
language into actions (Bisk et al., 2020a). We compose a Visual and Embodied
Concepts evaluation benchmark VEC, with examples shown in Table 1.

With the benchmark, we examine popular LMs including text-only LMs and vision-
augmented LMs. We cover masked language models and causal language models in
text-only LMs, including BERT (Devlin et al., 2019) and RoBERTa (Liu et al.,
2019b), and GPT (OPT)-family (Radford et al., 2019; Zhang et al., 2022b) with
parameters ranging from $125$M to $175$B. Furthermore, as humans understand
the world by learning from multiple modalities, especially using the visual
modality (Bloom, 2002), we are interested in whether the vision supervision in
recent vision-augmented language models (VLMs) (Chen et al., 2019; Radford et
al., 2021; Wang et al., 2021; Madureira, 2021) could also facilitate the
understanding ability of embodied concepts. CLIP (Radford et al., 2021) and
BLIP (Li et al., 2022a) are chosen as representatives of VLMs for evaluation,
due to their promising performance and the ability to deal with textual-only
inputs. To eliminate the effects of training corpus (Tan and Bansal, 2020), we
re-implement BERT, OPT, and CLIP on the same caption dataset with a similar
Transformer model (Vaswani et al., 2017) for a fair evaluation. Previous
studies have shown that prompting methods that fit the pre-training paradigm
could better elicit the knowledge learned from LMs (Petroni et al., 2019;
Schick and Schütze, 2021a; Brown et al., 2020). We adopt pre-trained-objective
style promoting methods to narrow the gap between probing and pre-training.

Our zero (few)-shot results on the VEC benchmark show that: (i) Moderate-sized
LMs such as BERT and RoBERTa exhibit a random-level understanding of both
visual and embodied concepts. (ii) A decent visual understanding of specific
concepts emerges as LMs scale up, while they still struggle to understand the
embodied knowledge with performance slightly better than random guessing.
(iii) Both image-grounded caption text and visual supervision could provide
performance gain regarding visual concepts, yet only the latter enhances the
understanding of embodied knowledge of LMs.

We further investigate the source of embodied knowledge in VLMs. A case study
demonstrates that embodied knowledge in the VLM of CLIP is potentially learned
from image representations. We thus propose a knowledge distillation method to
transfer the learned embodied knowledge in VLMs into LMs, resulting in an
average accuracy gain of $3.38$, which is comparable to the $4.46$ gain
achieved by scaling the model parameters $134$x. Nevertheless, the improved
LMs still exhibit great gaps with humans, indicating great potential for
further advancements.

##  2 VEC Benchmark

Our VEC benchmark aims to evaluate the understanding of physical concepts of
LMs. Inspired by the world scope definitions by Bisk et al. (2020a), we divide
physical knowledge into visual knowledge and embodied knowledge. The former
are visual properties that can be acquired via visual perception, while the
latter focus on knowledge that requires multimodal sensory interaction.

###  2.1 Visual Concepts

Perception is necessary for language learning because it forms the basis for
many of our semantic axioms (Bisk et al., 2020a). Among the various types of
perception, visual concepts model a vastness of experiences in the world that
cannot be stated by text alone (Harnad, 1990). In this work, we consider
evaluating the visual understanding ability of LMs by examining their
performance on various visual concepts. Specifically, we combine the recently
proposed visual knowledge probing datasets, including Spatial Commonsense (Liu
et al., 2022a) and ViComTe (Zhang et al., 2022a). The combined dataset
requires not only understanding various generic visual concepts including
color, shape, and material, but also understanding the relationship between
common objects, such as size and height. For generic visual concepts, i.e.,
color, shape, and material identification, we define an answer selection game:
selecting a correct value from two options for the attribute given an object.
For example, given a head object banana, the model should pick the ground-
truth tail answer yellow instead of an alternative option such as black. For
visual relationships, i.e., size and height understanding, we define a
comparison game: LMs need to perform a comparison between different objects.
For example, given a head entity ant and a tail entity bird, the LM is asked
to compare the size of two objects and makes a prediction between the correct
relation smaller and the false one larger.

###  2.2 Embodied Concepts

The embodied concepts refer to physical realities of objects, e.g., mass, and
temperature, which infants could learn by interacting with the environment
(Gopnik et al., 1999). This kind of knowledge is the basis of intelligence and
enables agent models to explore challenging tasks in physical environments. We
are curious about whether current LMs can capture embodied knowledge via
large-scale pre-training. In this work, we define embodied knowledge as the
knowledge that requires multimodal sensory interaction with the environments
beyond visual perception. We construct embodied knowledge evaluation datasets
regarding basic physical properties including mass, temperature, and hardness.

#### Mass Dataset

We build the Mass dataset by transforming the Image2Mass dataset curated by
Standley et al. (2017), which annotates common objects with corresponding
weights. The most light-weight object in the dataset is a red Lego brick,
weighing $0.026$ lbs, and the heaviest object is a $2.664$ lbs drill. Directly
asking the LM for the absolute mass of objects can be challenging (Wallace et
al., 2019). We define the task in a comparison format. Specifically, each
comparison pair contains two objects with a weight gap greater than $1$ lbs.
The threshold is set according to the Weber–Fechner laws (Fechner et al.,
1966) to guarantee that the mass difference is perceivable for humans. We
build $654$ triplets such as (hair dryer, heavier than, red Lego brick) for
evaluation.

#### Temperature Dataset

We design a temperature probing dataset by collecting the temperature of
common objects from
Wikipedia.222https://en.wikipedia.org/wiki/Orders_of_magnitude_(temperature)
For example, the ice is $0^{\circ}$C, and the temperature of water vapor is
$100^{\circ}$C. We convert the object with temperature annotations into pairs,
and each pair contains two objects and the corresponding temperature relation.
For example, (ice, colder than, water vapor). The temperature gap between two
objects must be greater than a difference threshold, which is loosely set to
$10^{\circ}$C for assurance of thermal perception for human (Jones, 2009). The
final Temperature dataset consists of $422$ pairs in total.

#### Hardness Dataset

Hardness is a measure of the resistance to localized plastic deformation in
material science. For example, hard metals such as titanium are harder than
soft minerals such as talc. Humans can perceive the hardness of different
materials in interaction with the environment by using tactile organs like
fingers (Gueorguiev et al., 2016). To investigate whether LMs capture hardness
knowledge, we build a Hardness dataset by collecting the Mohs hardness scores
of different objects from
Wikipedia.333https://en.wikipedia.org/wiki/Mohs_scale_of_mineral_hardness We
define the task in a comparison format. For example, (talc, softer than,
titanium). Each pair contains two objects. The gap between two objects is
greater than the threshold for human-level understanding. The final dataset
contains $1,016$ pairs.

Figure 1: An illustration of prompting methods. For BERT-like models with a
masked language head, we convert the knowledge fact to a question and perform
prediction with the head over _yes_ or _no_. For OPT models, we evaluate the
perplexity of different assertions and take the one with lower perplexity as a
valid fact. For CLIP, we devise a matching-based probing framework.

##  3 Prompting Methods

Recent studies have shown that prompting methods that fit the pre-training
paradigm are more effective than other possible prompting methods (Petroni et
al., 2019; Schick and Schütze, 2021a). Following these studies, we design
specific prompts for LMs with different objectives.

#### Prompting Masked Language Models

Following PET (Schick and Schütze, 2021a; Schick and Schütze, 2021b), we probe
the masked language models by converting knowledge facts into a question-
answering form. For example, a size knowledge fact (coin, smaller than, table)
is converted into a sentence with a special mask token: Question: is a coin
smaller than a table? Answer: [MASK]. We also explored other prompts, such as
Is a coin [MASK] than table. However, our experiments show that a question-
answering form can better induce models to generate answers and avoid the
influence of tokenization of different LMs. Given masked inputs, the model is
asked to predict the probabilities of the mask token over two choices, i.e.,
yes for confirming the knowledge fact is valid or no for an unreasonable
assertion. We observe that in specific LMs, the prediction can be biased
toward some answers as investigated by Zhao et al. (2021). We calibrate the
prediction by normalizing the probabilities according to an estimated prior
following Zhao et al. (2021).

Model (# of Param.) | Color | Shape | Size | Height | Material | Avg.
---|---|---|---|---|---|---
BERT${}_{\text{YFCC-15M}}$ (63M) | 56.05±10.36 | 53.21±1.79 | 50.34±1.27 | 50.16±1.30 | 55.35±4.79 | 53.02
OPT${}_{\text{YFCC-15M}}$(63M) | 65.21±15.27 | 51.25±18.99 | 50.50±0.77 | 49.96±1.36 | 81.41±1.53 | 59.67
CLIP${}_{\text{YFCC-15M}}$ (63M) | 68.21±7.17 | 67.21±7.63 | 62.64±6.01 | 54.04±7.05 | 62.92±6.48 | 63.00
BERT-base (110M) | 49.29±1.60 | 52.14±4.22 | 49.94±0.80 | 50.56±0.59 | 48.08±2.74 | 50.00
BERT-large (340M) | 49.36±1.88 | 51.21±5.06 | 49.26±1.60 | 49.08±2.34 | 49.72±0.58 | 49.73
RoBERTa-base (125M) | 49.07±1.62 | 49.36±3.52 | 50.32±0.57 | 49.58±0.49 | 49.86±1.44 | 49.64
RoBERTa-large (355M) | 49.66±0.54 | 50.68±1.48 | 50.54±1.46 | 50.14±0.45 | 50.00±0.14 | 50.20
OPT (125M) | 70.02±9.59 | 57.32±6.46 | 45.98±4.23 | 56.76±1.36 | 82.43±2.20 | 62.50
OPT (1.3B ) | 76.92±5.97 | 65.00±6.12 | 51.12±2.66 | 57.82±4.46 | 85.63±3.49 | 67.30
OPT (13B) | 79.62±5.28 | 62.50±6.44 | 57.56±6.60 | 54.58±4.53 | 88.38±3.14 | 68.53
OPT (175B) | 83.10±3.13 | 65.71±7.54 | 59.18±9.05 | 55.84±5.33 | 85.49±2.01 | 69.87
CLIP-ViT/B-32 (63M) | 80.07±2.57 | 84.43±2.57 | 61.40±6.02 | 62.28±6.40 | 80.07±2.57 | 73.94
DeCLIP-ViT/B-32 (63M) | 81.48±2.63 | 84.07±2.34 | 76.92±1.81 | 68.12±2.15 | 81.48±2.63 | 78.35
CLIP-ViT/L-14 (123M) | 80.33±3.61 | 85.00±4.03 | 63.96±6.10 | 60.72±5.56 | 80.33±3.61 | 74.21
BLIP-base (138M) | 82.60±5.50 | 84.86±2.80 | 76.00±6.40 | 69.84±7.76 | 80.67±1.24 | 78.79
Table 2: Zero-shot probing results on visual datasets. Models with the
YFCC-15M subscript represents that these models are trained from scratch on
YFCC-15M data. Scaling OPT-family brings clear improvements on size and color
datasets. The scaling law fails on the height dataset.

#### Prompting Causal Language Models

Different from BERT, there is no special [MASK] token in causal language
models like GPT (Radford et al., 2019). Therefore, introducing a special token
would result in an inconsistency between pre-training and evaluation. To
remedy this, for each knowledge fact, we state it in natural sentences
according to prompting templates and evaluate the sentence perplexity as the
proxy metric. Specifically, for size-property evaluation, we convert it into a
valid knowledge assertion $s1=$ A coin is smaller than a table, and an invalid
one by replacing the relation with the antonym adjective $s2=$ A coin is
larger than a table. The sentence with lower perplexity is then chosen as the
predicted one. We evaluate the perplexity of each sentence
$s=(w_{0},w_{1},\cdots,w_{n})$ as:

$\text{PPL}(s)\\!=\\!P_{\mathcal{M}}(s)^{-\frac{1}{n}}\\!=\\!\sqrt[n]{\\!\prod_{k=1}^{n}\\!\frac{1}{P_{\mathcal{M}}\left(w_{k}\\!\mid\\!w_{0},\\!w_{1},\\!\ldots,\\!w_{k-1}\right)}}$

where $P_{\mathcal{M}}$ denotes the conditional word probability of the causal
language model to be probed and $n$ is the number of tokens in $s$. We compare
the perplexity $\text{PPL}(s_{1})$ and $\text{PPL}(s_{2})$ and choose the
sentence with lower PPL as a more valid assertion and calculate the prediction
accuracy accordingly.

#### Prompting Vision-augmented Language Models of CLIP

Unlike masked and causal language models with language head that supports word
predictions, the text encoder in CLIP only has one sentence representation
without any pre-trained language heads. To probe the learned knowledge in VLMs
of CLIP, we design a matching-based prompting method. In more detail, for the
size fact stated before, we first obtain two object descriptions $o_{1}=$ a
photo of a coin, and $o_{2}=$ a photo of a table. These two sentences are
encoded to get the corresponding object vectors via the CLIP language encoder:

${\mathbf{o}}_{1},{\mathbf{o}}_{2}=\text{CLIP}(o_{1}),\text{CLIP}(o_{2}).$

We then derive an attribute sentence $a=$ a photo of a small object, and
encode it to an attribute adjective vector with the language encoder:

${\mathbf{a}}=\text{CLIP}(a).$

The prediction is then performed by comparing the cosine similarity
$\text{cos}(\mathbf{o}_{1},\mathbf{a})$ and
$\text{cos}(\mathbf{o}_{2},\mathbf{a})$.444The matching-based prompting also
applies to the pooled embedding of BERT, yet the results exhibit great
variance as shown in Appendix A. The object with higher similarity with the
attribute description is adopted as the answer, i.e., a coin is smaller than a
table, if
$\text{cos}(\mathbf{o}_{1},\mathbf{a})>\text{cos}(\mathbf{o}_{2},\mathbf{a})$.
Otherwise, we assume that the model thinks the reversed relation holds. We can
also adopt the antonym adjective _large_ for getting the attribute vectors.
The results of the best-performing adjective words for CLIP are reported and
we discuss the influence of adjective options in § 4.3.

##  4 Experiments

Model (# of Param.) | Mass | Temperature | Hardness | Avg.
---|---|---|---|---
BERT${}_{\text{YFCC-15M}}$(63M) | 50.73±2.53 | 49.50±1.19 | 50.91±1.04 | 50.38
GPT${}_{\text{YFCC-15M}}$(63M) | 50.02±0.05 | 57.73±2.24 | 50.04±2.98 | 52.61
CLIP${}_{\text{YFCC-15M}}$(63M) | 67.45±5.16 | 64.83±4.17 | 62.22±3.11 | 64.83
BERT-base (110M) | 50.35±0.56 | 49.67±0.56 | 50.20±0.43 | 50.07
BERT-large (340M) | 49.97±1.31 | 49.83±0.50 | 49.98±0.06 | 49.93
RoBERTa-base (125M) | 49.65±0.51 | 50.00±0.00 | 48.04±2.04 | 49.23
RoBERTa-large (355M) | 50.08±0.23 | 50.07±0.19 | 49.95±0.15 | 50.03
OPT (125M) | 50.00±0.00 | 54.53±4.33 | 46.16±2.45 | 50.23
OPT (1.3B) | 50.05±0.10 | 50.90±5.08 | 53.03±2.69 | 51.33
OPT (13B) | 50.14±0.36 | 51.85±6.34 | 52.38±3.09 | 51.46
OPT (175B) | 50.21±0.24 | 59.83±8.68 | 57.33±3.41 | 55.79
CLIP-ViT/B-32 (63M) | 65.20±4.75 | 60.28±6.83 | 59.43±2.00 | 61.64
DeCLIP-ViT/B-32 (63M) | 54.95±2.00 | 68.58±3.08 | 61.10±4.14 | 61.54
CLIP-ViT/L-14 (123M) | 73.15±6.34 | 65.88±2.31 | 69.57±2.26 | 69.53
BLIP-base (138M) | 83.94±2.59 | 74.98±5.60 | 56.93±5.56 | 71.95

Table 3: Zero-shot results on embodied datasets. LMs struggle to understand
embodied knowledge, including OPT (175B) and visual-augmented LMs, with 71.95
as the best average performance.

###  4.1 Experimental Settings

#### Models

We cover two kinds of LMs, text-only LMs and visual-augmented LMs. Text-only
LMs include BERT-base (large) (Devlin et al., 2019), RoBERTa-base (large) (Liu
et al., 2019b) for masked language models, and OPT models with parameters
ranging from $125$M to $175$B. For VLMs, we include the text encoders of CLIP-
ViT-B/32 and CLIP-ViT-L/14 (Radford et al., 2021) as a base and a large
version, respectively. We also include an enhanced VLM with masked language
modeling as self-supervision, DeCLIP-ViT-B/32 (Li et al., 2022b) and BLIP, a
boosted VLM by unifying multi-modal understanding and generation tasks (Li et
al., 2022a).555https://huggingface.co/Salesforce/blip-itm-base-coco Since
directly comparing the VLMs and text-only LMs can be unfair due to the
difference in model configuration and training corpus (Tan and Bansal, 2020),
we re-train CLIP, BERT, and GPT from scratch with a similar Transformer model
on the same text corpus, the caption dataset in the YFCC-15M dataset (Thomee
et al., 2016). All models are trained for $32$ epochs. The only difference
between these models is the pre-training objective. Detailed model and
training settings are elaborated in Appendix B.

#### Prompts

We manually write several prompts (at least $4$ prompts for each task) to
eliminate the side-effect of the expression variations and report the averaged
accuracy. Besides, the variance across different prompts could also serve as
an indicator to evaluate the robustness of learned knowledge facts. We report
the averaged performance over multiple prompts for all models. All used
prompts can be found in Appendix C.

###  4.2 Main Findings

#### The ability of certain visual concepts emerges as scaling up LMs, but
there are still basic visual concepts where the scaling law fails.

The evaluation results on visual datasets are shown in Table 2. Interestingly,
with the scaling up of OPT-family models, the prediction accuracy increases
obviously on specific visual concepts such as color and size. On material and
color, the largest OPT-175B model even achieves better results than VLMs of
CLIP-ViT/L-14, which are augmented with vision supervision and are supposed to
perform better (Zhang et al., 2022a; Liu et al., 2022b). It is because the
combination of color and material frequently occurs (e.g., red apples) in raw
texts. The co-occurrence statistics of color and material are well captured by
large LMs. This is supported by the performance improvements after training on
visual-grounded text corpus YFCC-15M. However, we also notice that increasing
LMs to 175B brings negligible improvements in the Height dataset, indicating
that there still remain visual concepts where the scaling law does not hold
even though these concepts can be easily captured by humans.

Model | Mass | Temperature | Hardness | Avg.
---|---|---|---|---
Zero-shot Best VLMs | 83.94±2.59 | 74.98±5.60 | 69.57±2.26 | 76.16
BERT-base | 64.72±4.77 | 55.62±1.34 | 51.80±1.31 | 57.38
BERT-large | 65.47±4.86 | 54.19±2.31 | 52.73±1.22 | 57.46
RoBERTa-base | 60.24±5.24 | 60.27±4.85 | 50.44±1.13 | 56.98
RoBERTa-large | 61.18±4.01 | 58.28±2.09 | 50.56±1.14 | 56.67

Table 4: The few-shot results of BERT variants. With $16$ instances, the fine-
tuned BERT variants are still worse than zero-shot visual-augmented LMs.

#### LMs exhibit a poor understanding of embodied concepts.

As shown in Table 3, the scaling law fails again on the embodied concepts, as
all LMs, including OPT-175B and variants trained with captions data, perform
poorly. Although VLMs achieve better results than text-only LMs, they still
show poor results, slightly better than random guessing on all three embodied
tasks. We further conduct a few-shot prompt evaluation for OPT models by
constructing the inputs with $k=16$ randomly sampled instances and adopt PET
(Schick and Schütze, 2021a) for masked language models. The results are
illustrated in Figure 2 and Table 4, respectively. We find that while the
performance is boosted, the average results are still worse than the best-
performing VLM model without any demonstration, which only utilizes $0.08\%$
parameters of OPT-175B. These findings show that visual supervision can help
learn embodied knowledge, but there is still a large gap between the best
results of existing LMs with human performance.

Figure 2: Few-shot results of OPT-175B with $16$ instances as demonstration on
embodied tasks.

#### Compared with human annotators, OPT-175B and VLMs achieve competitive
performance regarding visual concepts, yet exhibit great gaps with humans on
embodied concepts.

We conduct a human evaluation to better understand the performance of
different models. Specifically, we randomly sample $100$ examples for each
task and ask three volunteers to label these examples. The annotators achieve
substantial agreements on all the tasks with Cohen’s kappa (Cohen, 1960)
$\kappa$ larger than $0.7$, except for the Hardness dataset with a moderate
$\kappa=0.52$. The comparison with best-performing models, i.e., OPT-175B,
CLIP-ViT/L-14 and DeCLIP is illustrated in Figure 3. We find that (i)
Regarding visual concepts, both OPT and CLIP-like models perform closely to
human annotators. CLIP and DeCLIP even outperform the human annotators on the
shape task, which is potentially due to the noise introduced by the automatic
construction of the dataset (Zhang et al., 2022a). Overall, the close-to-human
results indicate that visual knowledge can be effectively acquired by large-
scale cross-modal pre-training or even text-only pre-training with sufficient
parameters. (ii) Regarding embodied concepts, the best-performing CLIP-
ViT-L/14 model still has an absolute $18.5\%$ accuracy gap with the humans.
The clear performance gaps reveal that there is still a long way to go in
equipping LMs and VLMs with embodied knowledge.

Figure 3: Comparison between the best-performing models and human annotators
on sampled subsets of VEC. The best-performing LMs and VLMs achieve close-to-
human results on visual datasets, yet far lag behind humans in embodied
datasets.

###  4.3 Analysis

#### Does BERT behave similarly regarding visual and embodied concepts?

The overall prediction results of BERT-like models in the visual and embodied
world are both at a random level. We investigate this question result by first
checking whether BERT models perform consistently at a guessing level for all
the entities in the dataset. We compute the entity correct ratio among
different prompts for the objects in different datasets and compare the
distribution on different tasks with the BERT model trained on YFCC-15M
dataset. As illustrated in Figure 4, in the Material identification task,
there are entities that the model that could provide consistent correct
predictions. However, the distribution of the Hardness dataset in embodied
evaluation exhibits a bell curve, i.e., most entities are predicted correctly
at a random-chance level. The distributions of other tasks show similar
results and can be found in Appendix D. These results suggest that BERT learns
visual knowledge for certain entities yet indeed struggles regarding embodied
concepts.

Figure 4: Entity correct ratio histograms of Mass and Material datasets across
different prompts. BERT could make consistent correct predictions for specific
entities, and the bell curve on the hardness indicates it is challenging for
BERT to understand embodied concepts.

#### Learned embodied knowledge in image representations

We are interested in how the VLMs of CLIP learn embodied knowledge. A
potential answer is that the images contain rich semantics regarding embodied
knowledge such as the heat of the object, and the knowledge can be propagated
to the VLMs via the contrastive learning objective. To examine this, we
perform a case study by calculating the attribute similarities among the
images. We first take clips from a video of heating a pile of ice and then
perform a binary classification by calculating the cosine similarities with
text prompts _a photo of a hot object._ and _a photo of a cold object_ for
each frame. The left of Figure 5 shows that the probability of a hot object
increases during the heating procedure. Similarly, we perform a binary
classification over heavy and light-weight objects ranging from an elephant to
a feather. The right of Figure 5 shows that the image representations are
aware of the mass of different objects. This qualitative study shows that
image representations are the potential source of embodied knowledge.

#### Transferring embodied knowledge from VLMs to LMs

We further verify whether the learned embodied knowledge in CLIP could be
transferred to text-only models. Specifically, we perform knowledge
distillation (Hinton et al., 2015) by treating the original text-only language
model as a student, and the CLIP text encoder as a teacher model providing the
learned embodied knowledge. However, our preliminary study in Appendix F shows
that vanilla alignment on the predicted word distributions could not be
effective. Inspired by our case study showing that the rich embodied knowledge
contained in the representations, we utilize Neuron Selectivity Transfer
(Huang and Wang, 2017) which transfers the inner states such as spatial
activation patterns of teacher neurons to student neurons, by aligning the
token representations of the last layer between the teacher and student
language models, which is implemented as a squared maximum mean discrepancy
(MMD) with a polynomial kernel to measure the distance between the activation
patterns of student neurons and teacher neurons. The total training objective
of the language model is a combination of the original language modeling loss
and the MMD loss with a balancing coefficient. We refer readers to Appendix E
for more details. As shown in Table 5, the distillation provides a performance
boost on embodied concepts understanding, e.g., learning from a CLIP-ViT-L/14
teacher model achieves improvement that is comparable with that brought by
scaling the model parameter $134$x from OPT-1.3B to OPT-175B.666Only OPT is
adopted for experiments as the CLIP encoder cannot deal with the mask tokens
introduced in BERT. It validates our assumption and indicates that future
studies could utilize the richer representations in VLMs for improving LMs,
yet the gap between distilled LM and VLM suggests that there is still room for
advancement.

Figure 5: Case study showing that the image representations in CLIP exhibit
embodied knowledge. (Left) The probability of an image being classified as
"hot" increases as the ice melts being heated in a boiler in a video. (Right)
The probability of an image being classified as "heavy" along with
corresponding mass annotation.

#### VLMs perform poorly when dealing with ambiguous text descriptions

Figure 6: Top-5 retrieved images and the prediction accuracy with different
attribute prompts. The accuracy drops when the text inputs contain ambiguous
words and compound words, as the retrieved images are biased toward specific
meanings.

During our preliminary study, we observe that VLMs of CLIP perform relatively
poorly for specific adjectives such as _hard_. To further investigate this
issue, we examine the retrieved images using prompts with different attribute
adjectives on the CC12M dataset (Changpinyo et al., 2021). Our results,
illustrated in Figure 6, revealed that for the prompt _a photo of a hard
object_ , the retrieved images were mostly about abstract and difficult
learning materials, with only one rock image related to the attribute of
hardness. Additionally, for the prompt _light-weight_ , the retrieved images
are biased towards meanings related to lighting bulbs and light-toned colors.
These observations demonstrate that handling semantic ambiguity remains a
challenge for VLMs, suggesting that future improvements may incorporate more
language-side supervision such as DeCLIP.

##  5 Related Work

#### Probing Language Models

Understanding what LMs know after large-scale pre-training is an active
research area (Rogers et al., 2020). Various probing methods have been
developed (Tenney et al., 2019b; Petroni et al., 2019), and investigations
show that LMs capture linguistic (Tenney et al., 2019a; Liu et al., 2019a),
factual (Petroni et al., 2019; Roberts et al., 2020; Dai et al., 2022),
commonsense knowledge (Wang et al., 2019; Forbes et al., 2019), and even
acquire grounded concepts (Patel and Pavlick, 2022). For VLMs, studies
demonstrate their potential in acquiring spatial commonsense (Zhang et al.,
2022a; Liu et al., 2022a) and color perception (Abdou et al., 2021), yet
performing worse on NLU tasks (Tan and Bansal, 2020) and achieving no
significant on lexical grounding (Yun et al., 2021). In this paper, we
investigate the understanding ability of LMs on physical concepts. Different
from PIQA (Bisk et al., 2020b) consists of questions requiring physical
commonsense reasoning, our VEC benchmark examines the understanding ability of
the most fundamental physical concepts. The evaluation on the VEC benchmark
demonstrates that text-only LMs can learn specific visual concepts after
scaling up while struggling with the embodied concepts.

Model | Mass | Temperature | Hardness | Avg.
---|---|---|---|---
CLIP-ViT/B-32 (T1) | 65.20±4.75 | 60.28±6.83 | 59.43±2.00 | 61.64
CLIP-ViT/L-14 (T2) | 73.15±6.34 | 65.88±2.31 | 69.57±2.26 | 69.53
OPT-1.3B | 50.05±0.10 | 50.90±5.08 | 53.03±2.69 | 51.33
scale up to 13B | 50.14±0.36 | 51.85±6.34 | 52.38±3.09 | 51.46 (+0.13)
scale up to 175B | 50.21±0.24 | 59.83±8.68 | 57.33±3.41 | 55.79 (+4.46)
OPT${}_{\text{YFCC-15M}}$ | 50.02±0.05 | 57.73±2.24 | 50.04±2.98 | 52.61
Distill w/ T1 | 49.88±0.37 | 55.76 ±4.01 | 53.23±3.12 | 52.96 (+0.35)
Distill w/ T2 | 54.27±5.20 | 60.78±4.23 | 52.91±1.62 |  55.99 (+3.38)

Table 5: Results of embodied distillation. Transferring embodied knowledge
from CLIP-ViT to OPT brings a gain of 3.38 points, which is comparable with
the improvements by scaling the model from 1.3B to 175B.

#### Vision-Language Pre-training

Unifying cross-modal representations via vision-language pre-training has
achieved promising progress. Pilot studies adopt masked reconstruction to
learn shared representations across modalities from a mixed visual and
language inputs (Li et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Chen
et al., 2019; Li et al., 2020). CLIP (Radford et al., 2021) introduces a
contrastive language-image pre-training framework, utilizing language as
supervision for learning transferable image representations with large-scale
image-text pairs, triggering a series of variants for further improvements
(Jia et al., 2021; Li et al., 2022b; Yao et al., 2021; Li et al., 2021,
2022a). Our study uses VLMs of CLIP and its variants to investigate the impact
of visual supervision on understanding physical concepts and our results
suggest that visual supervision is crucial for LMs to understand embodied
concepts.

##  6 Conclusion

In this paper, we introduce VEC for evaluating the understanding of physical
concepts in LMs. Our results show that large LMs understand specific visual
concepts but struggle with embodied knowledge. VLMs instead perform much
better in both the visual and the embodied world, indicating that visual
signals are vital for understanding physical concepts. Further analysis
suggests that transferring the VLM representations to LMs is effective for
boosting embodied concepts understanding.

## Limitations

We evaluate three physical properties, including mass, temperature, and
hardness in this work. These aspects are selected as they could be measured by
well-established metrics and easily sensed by humans. While this is a biased
approximation, it is still sufficient to demonstrate that the current text-
only LMs perform poorly regarding embodied knowledge and suggest that vision
supervision could help understand embodied concepts. We will dynamically
update our benchmark by incorporating new tasks in the future.

While there exist many multi-modal models, we only adopt VLMs of CLIP and its
variants for our investigation. We select CLIP due to its superior image
representation performance and support text-only encoders. As our evaluation
is language-oriented, the evaluated models are supposed to be able to deal
with inputs with pure text. VLMs such as UNITER (Chen et al., 2019) only
support multi-modal inputs and thus cannot be selected. Therefore, CLIP is
chosen as a representative work of VLMs for evaluation. However, the findings
of CLIP may not transfer to other V+L models trivially, as CLIP utilizes
million-level image-text pairs collected from the web, which could also become
a significant source of embodied knowledge. Besides, there are recently
proposed VLMs models with various architectures and pre-training recipes such
as SimVLM (Wang et al., 2021), UniT (Hu and Singh, 2021), ViLT (Kim et al.,
2021) and FLAVA (Singh et al., 2022), which show promising performance in both
cross-modal and single-modality tasks. We will examine these advanced models
in our benchmark in the future.

## References

  * Abdou et al. (2021) Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021.  Can language models encode perceptual structure without grounding? a case study in color.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 109–132, Online. Association for Computational Linguistics.
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_.
  * Bisk et al. (2020a) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020a.  Experience grounds language.  In _Proc. of EMNLP_.
  * Bisk et al. (2020b) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020b.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 7432–7439.
  * Bloom (2002) Paul Bloom. 2002.  _How children learn the meanings of words_.  MIT press.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _CVPR_.
  * Chen et al. (2019) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019.  Uniter: Learning universal image-text representations.  _ArXiv_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Knowledge neurons in pretrained transformers.  In _Proc. of ACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proc. of NAACL_.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023\.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_.
  * Fechner et al. (1966) Gustav Theodor Fechner, Davis H Howes, and Edwin Garrigues Boring. 1966.  _Elements of psychophysics_ , volume 1.  Holt, Rinehart and Winston New York.
  * Forbes et al. (2019) Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019.  Do neural language representations learn physical commonsense?  In _CogSci_.
  * Gopnik et al. (1999) Alison Gopnik, Andrew N Meltzoff, and Patricia K Kuhl. 1999.  _The scientist in the crib: Minds, brains, and how children learn._ William Morrow & Co.
  * Gueorguiev et al. (2016) David Gueorguiev, Séréna Bochereau, André Mouraux, Vincent Hayward, and Jean-Louis Thonnard. 2016.  Touch uses frictional cues to discriminate flat materials.  _Scientific reports_.
  * Hao et al. (2022) Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. 2022.  Language models are general-purpose interfaces.  _arXiv preprint arXiv:2206.06336_.
  * Harnad (1990) Stevan Harnad. 1990.  The symbol grounding problem.  _Physica D: Nonlinear Phenomena_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Hu and Singh (2021) Ronghang Hu and Amanpreet Singh. 2021.  Unit: Multimodal multitask learning with a unified transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1439–1449.
  * Huang and Wang (2017) Zehao Huang and Naiyan Wang. 2017.  Like what you like: Knowledge distill via neuron selectivity transfer.  _arXiv preprint arXiv:1707.01219_.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017.  Quora question pairs.  _First Quora Dataset Release: Question Pairs_.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proc. of ICML_.
  * Jones (2009) Lynette Jones. 2009.  Thermal touch.  _Scholarpedia_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR.
  * Li et al. (2022a) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR.
  * Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  In _Proc. of Neurips_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _ArXiv_.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Proc. of ECCV_.
  * Li et al. (2022b) Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. 2022b.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  In _Proc. of ICLR_.
  * Liu et al. (2019a) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a.  Linguistic knowledge and transferability of contextual representations.  In _Proc. of NAACL_.
  * Liu et al. (2022a) Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022a.  Things not written in text: Exploring spatial commonsense from visual signals.  In _Proc. of ACL_.
  * Liu et al. (2022b) Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022b.  Things not written in text: Exploring spatial commonsense from visual signals.  _ArXiv preprint_.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: A robustly optimized BERT pretraining approach.  _ArXiv preprint_.
  * Madureira (2021) Brielen Madureira. 2021.  Flamingos and hedgehogs in the croquet-ground: Teaching evaluation of NLP systems for undergraduate students.  In _Proceedings of the Fifth Workshop on Teaching NLP_ , pages 87–91, Online. Association for Computational Linguistics.
  * Patel and Pavlick (2022) Roma Patel and Ellie Pavlick. 2022.  Mapping language models to grounded conceptual spaces.  In _International Conference on Learning Representations_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proc. of EMNLP-IJCNLP_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proc. of ICML_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proc. of EMNLP_.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model?  In _Proc. of EMNLP_.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in BERTology: What we know about how BERT works.  _Transactions of the Association for Computational Linguistics_.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proc. of EACL_.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Singh et al. (2022) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15638–15650.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proc. of EMNLP_.
  * Standley et al. (2017) Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. 2017.  image2mass: Estimating the mass of an object from its image.  In _Proceedings of the 1st Annual Conference on Robot Learning_.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  VL-BERT: pre-training of generic visual-linguistic representations.  In _Proc. of ICLR_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _Proc. of EMNLP-IJCNLP_.
  * Tan and Bansal (2020) Hao Tan and Mohit Bansal. 2020.  Vokenization: Improving language understanding with contextualized, visual-grounded supervision.  In _Proc. of EMNLP_.
  * Tenney et al. (2019a) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.  BERT rediscovers the classical NLP pipeline.  In _Proc. of ACL_.
  * Tenney et al. (2019b) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _Proc. of ICLR_.
  * Thomee et al. (2016) Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wallace et al. (2019) Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019.  Do NLP models know numbers? probing numeracy in embeddings.  In _Proc. of EMNLP-IJCNLP_.
  * Wang et al. (2019) Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019.  Does it make sense? and why? a pilot study for sense making and explanation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4020–4026, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021\.  Simvlm: Simple visual language model pretraining with weak supervision.  In _International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proc. of ACL_.
  * Yao et al. (2021) Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021.  Filip: Fine-grained interactive language-image pre-training.  _ArXiv preprint_.
  * Yun et al. (2021) Tian Yun, Chen Sun, and Ellie Pavlick. 2021.  Does vision-and-language pretraining improve lexical grounding?  In _Proc. of EMNLP Findings_.
  * Zhang et al. (2022a) Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel-Eskin. 2022a.  Visual commonsense in pretrained unimodal and multimodal models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5321–5335, Seattle, United States. Association for Computational Linguistics.
  * Zhang et al. (2022b) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022b.  Opt: Open pre-trained transformer language models.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _Proc. of ICML_.

##  Appendix A Matching-based Prompting Results for BERT

We perform the identical prompting method and templates with CLIP VLMs for LMs
of BERT. RoBERTa is discarded as there is no pooled embedding during its pre-
training for representing the sentence. As shown in Table 9, the performance
is still close to a random guessing baseline. Besides, the higher variances on
all the tasks compared to VLMs of CLIP indicate that this method does not fit
LMs of BERT well, validating our prompting design choice in the main paper to
fit the pre-training paradigm.

##  Appendix B Model Configurations

Model | Hidden Layers | Hidden Size | Attention Heads | Total # of Parameters
---|---|---|---|---
BERT-base | 12 | 768 | 12 | 110M
BERT-large | 24 | 1,024 | 16 | 340M
RoBERTa-base | 12 | 768 | 12 | 125M
RoBERTa-large | 24 | 1,024 | 16 | 355M
OPT-125M | 12 | 768 | 12 | 125M
OPT-1.3B | 24 | 2,048 | 32 | 1.3B
OPT-13B | 40 | 5,120 | 40 | 13B
OPT-175B | 96 | 12,288 | 96 | 175B
CLIP-ViT/B-32 | 12 | 512 | 8 | 63M
DeCLIP-ViT/B-32 | 12 | 512 | 8 | 63M
CLIP-ViT/L-14 | 12 | 768 | 12 | 123M

Table 6: Detailed configuration of models evaluated in the paper.

Here we provide the detailed configurations of evaluated LMs and VLMs in our
main paper. For the vanilla model, there model configurations are listed in
Table 6. For the models pre-trained from scratch with the YFCC-15M dataset,
they are adopted the same LM architecture as the VLM of CLIP-ViT/B-32, and the
only difference is the pre-training objective, as shown in Table 7. All these
models are optimized using an Adam optimizer with a learning rate set to
$1$e$-4$, linearly increased at the first $2000$ steps. The batch size is
$2048$ and all models are trained with $32$ epochs. We use 1% of the data for
evaluation, and the final OPT-YFCC15M model gets a $31.9$ validation
perplexity.

Model | Training Objective | Training Dataset
---|---|---
BERT${}_{\text{YFCC-15M}}$ | Masked Language Modeling (MLM) | Captions in YFCC-15M
GPT${}_{\text{YFCC-15M}}$ | Causal Language Modeling (MLM) | Captions in YFCC-15M
CLIP${}_{\text{YFCC-15M}}$ | Contrastive Image-text Matching (CIM) | Image-Text Pair in YFCC-15M

Table 7: Pre-training objectives and corpus comparison of YFCC-15M models
evaluated in the main paper.

##  Appendix C Details of Prompts

We provide the used prompts for evaluating different models based on their
pre-training objectives. Examples of Head, Rel and Tail of each dataset are
shown in Table 1. Due to the sensitivity of language models to prompts, we
provide diverse prompts for each model on each task.

#### Prompts for Masked Language Models

A [MASK] token is placed in the prompt and the models are asked to predict the
probabilities of the [MASK] token. To avoid multiple mask tokens in prompts,
we follow Schick and Schütze to convert knowledge fact into a cloze-question.
For example, a temperature fact (water, colder than, frying oil) can be
converted into Q: is the water colder than frying oil? A: [MASK]!. The models
need to choose the token yes or no to fill the mask.

#### Prompts for Causal Language Models

As there is no special [MASK] token during the pre-training of causal language
models, we do not use [MASK] tokens in prompts for causal language models. For
Color, Shape and Material datasets of the visual concepts we construct two
prompts for (Head, Tail1) and (Head, Tail2); while for other datasets, we
construct two prompts for (Head, Rel, Tail) and (Head, Rel′, Tail) where Rel′
is the antonym relation of Rel. The prediction is based on the prompt with
lower perplexity.

#### Prompts for CLIP

Following Radford et al. (2021), we use prompts like a photo of ... here. As
the language encoder of CLIP encodes sentences to a vector and can evaluate
similarities between sentences. We use an attribute prompt like a photo of a
cold object and construct same prompts for objects (water and frying oil) in
the knowledge fact. We can determine the colder object if the prompt of this
object has a higher similarity to the attribute prompt.

Model | SST-2 | QQP | QNLI | MNLI (m / mm) | Avg.
---|---|---|---|---|---
BERT (Wiki) | 90.13 | 83.20 | 87.57 | 78.90 / 80.05 | 83.97
DistilledOscar | 89.33 | 67.98 | 82.48 | 74.46 / 74.82 | 77.81
VLM-BERT-base | 90.60 | 90.10 | 89.47 | 81.57 / 82.43 | 86.83
VLM-RoBERTa-base | 90.13 | 88.44 | 87.91 | 80.37 / 80.43 | 85.46
Model | Color | Shape | Size | Height | Material | Mass | Temperature | Hardness | Avg.
BERT (Wiki) | 49.41 | 48.07 | 51.70 | 49.46 | 52.39 | 48.85 | 51.07 | 52.34 | 50.41
DistilledOscar | 49.97 | 53.61 | 49.07 | 49.80 | 51.46 | 51.22 | 47.94 | 51.23 | 50.54
VLM-BERT-base | 50.69 | 50.07 | 51.00 | 50.92 | 53.89 | 44.83 | 50.64 | 49.22 | 50.16
VLM-RoBERTa-base | 49.53 | 51.21 | 49.00 | 49.22 | 49.54 | 49.92 | 51.11 | 49.63 | 49.90

Table 8: Fine-tuned accuracy of other visual-informed pre-trained language
models on NLU tasks and zero-shot results regarding the physical concepts.

Model | Color | Shape | Size | Height | Material | Mass | Temperature | Hardness | Avg.
---|---|---|---|---|---|---|---|---|---
CLIP-ViT/L-14 | 80.33±3.61 | 85.00±4.03 | 63.96±6.10 | 60.72±5.56 | 80.33±3.61 | 73.15±6.34 | 65.88±2.31 | 69.57±2.26 | 72.37
BERT-base Pooled | 43.19±5.13 | 59.64±7.24 | 66.10±7.91 | 65.48±6.63 | 46.55±6.18 | 52.32±7.46 | 56.59±5.65 | 55.51±5.50 | 55.67
BERT-large Pooled | 44.74±5.93 | 56.93±6.45 | 53.80±5.92 | 54.84±8.01 | 52.18±4.88 | 57.92±7.26 | 51.90±6.56 | 56.22±4.45 | 53.57

Table 9: Zero-shot results of BERT models with pooled output as sentence
embedding on VEC benchmark.

##  Appendix D Entity Analysis

In our main paper, we investigate the random-level performance of BERT models
by exploring the correct ratio over different prompts. We provide full
histogram plots of all tasks in Figure 7, 8, 9,10, 11, 12, 13, and 14. It can
be found that for visual concepts tasks such as material and shape, there are
entities that BERT could produce consistent correct prediction across
different prompts. However, for all embodied tasks, the histograms exhibit
bell curves, indicating the poor understanding ability of BERT on embodied
concepts.

##  Appendix E Embodied Knowledge Transfer

We provide implementation details here for the knowledge transfer experiments
from VLMs to LMs. Specifically, we take the VLM as a teacher model $T$ (e.g.,
the text encoder of the CLIP model) and the LM as a student model $S$ (e.g.,
the OPT language model). Given a text $\mathbf{x}$ from the training dataset
$\mathcal{D}$, we transfer the sequential activation patterns of
$T(\mathbf{x})\in R^{|x|\times d}$ to $S(\mathbf{x})\in R^{|x|\times d}$ ,
where $T(\mathbf{x})$ and $S(\mathbf{x})$ denote the last hidden
representations of the VLM and the LM, respectively. $d$ is the number of
hidden units. The squared maximum mean discrepancy (MMD) with kernel trick
(Huang and Wang, 2017) is adopted to measure the distance between the
activation patterns:

$\displaystyle\operatorname{MMD}^{2}(\mathbf{x})=$
$\displaystyle\frac{1}{d^{2}}\sum_{i=1}^{d}\sum_{i^{\prime}=1}^{d}k\left[S(\mathbf{x})_{*,i};S(\mathbf{x})_{*,i^{\prime}}\right]$
$\displaystyle+\frac{1}{d^{2}}\sum_{j=1}^{d}\sum_{j^{\prime}=1}^{d}k\left[T(\mathbf{x})_{*,j};T(\mathbf{x})_{*,j^{\prime}}\right]$
$\displaystyle-\frac{2}{d^{2}}\sum_{i=1}^{d}\sum_{j=1}^{d}k\left[S(\mathbf{x})_{*,i};T(\mathbf{x})_{*,j}\right]$

We adopt a polynomial kernel
$k(\mathbf{x};\mathbf{y})=\left(\mathbf{x}^{\top}\mathbf{y}+c\right)^{p}$ with
$p=2$ and $c=0$. The MMD objective $\mathcal{L}_{\text{MMD}}$ is minimized
along with the original language modeling objective $\mathcal{L}_{\text{LM}}$:

$\mathcal{L}=\mathcal{L}_{\text{lm}}+\beta\mathcal{L}_{\text{MMD}}$

where $\beta$ is a weighting factor set to $20$ to achieve a balance between
objectives.

##  Appendix F Evaluation and Distillation with Oscar

We examine whether other vanilla distillation from traditional V+L pre-
training methods brings gains regarding visual and embodied knowledge.
Specifically, following Zhang et al. (2022a), we distill the knowledge of
Oscar (Li et al., 2020) into a BERT model by performing knowledge distillation
(Hinton et al., 2015) on the image-caption pair dataset. Specifically, the
paired text and image are fed into the Oscar model for getting the vision-
aware vocabulary distribution, and a student BERT model is performing masked
language modeling on the text data only and learns from the soft labels
provided by the Oscar teacher model. The distillation results in a
DistilledOscar model supporting text-only inputs. We also evaluate VLM-BERT
learned via Vokenziation (Tan and Bansal, 2020), which devises a fine-grained
token-voken matching framework to utilize visual supervision. The models are
evaluated on the four largest datasets in GLUE, including SST-2 (Socher et
al., 2013), QQP (Iyer et al., 2017), QNLI (Rajpurkar et al., 2016) and MNLI
(Williams et al., 2018) for stable results. As shown in Table 8,
DistilledOscar performs worse than the vanilla BERT in both NLU tasks and
probing tasks regarding visual and embodied knowledge. Besides, while VLM-BERT
achieves improvements on NLU tasks, it still performs at the random level on
the probed tasks. These indicate that not all VLMs could learn embodied
knowledge and it is non-trivial to distill the visual supervision from VLMs to
LMs via purely language modeling.

Figure 7: Histogram of entity correct ratio across different prompts on the Size dataset. Figure 8: Histogram of entity correct ratio across different prompts on the Shape dataset. Figure 9: Histogram of entity correct ratio across different prompts on the Color dataset. Figure 10: Histogram of entity correct ratio across different prompts on the Height dataset. Figure 11: Histogram of entity correct ratio across different prompts on the Material dataset. Figure 12: Histogram of entity correct ratio across different prompts on the Temperature dataset. Figure 13: Histogram of entity correct ratio across different prompts on the Hardness dataset. Figure 14: Histogram of entity correct ratio across different prompts on the Mass dataset. Table 10: Prompts for Masked Language Models Model | Task | Prompt
---|---|---
BERT & RoBERTa | Size, Height, Temperature, Weight, Hardness |  | is the [Head] [Rel] than the [Tail]? [MASK]!
---
is the [Head] [Rel] than the [Tail]? [MASK].
is [Head] [Rel] than [Tail]? [MASK]!
is [Head] [Rel] than [Tail]? [MASK].
is [Head] [Rel] compared with [Tail]? [MASK].
is [Head] [Rel] compared with [Tail]? [MASK]!
compared with [Tail], is [Head] [Rel]? [MASK].
compared with [Tail], is [Head] [Rel]? [MASK]!
is [Head] usually [Rel] than [Tail]? [MASK].
is [Head] usually [Rel] than [Tail]? [MASK]!
Color |  | can [Head] be of color [Tail]? [MASK]!
---
can [Head] be of color [Tail]? [MASK].
is the color of a [Head] [Tail]? [MASK]!
is the color of a [Head] [Tail]? [MASK].
is [Head] [Tail]? [MASK].
is [Head] [Tail]? [MASK]!
is [Head] typically in [Tail]? [MASK].
is [Head] typically in [Tail]? [MASK]!
Q: is [Head] of color [Tail]? A: [MASK].
Question: is [Head] of color [Tail]? Answer: [MASK].
Shape |  | can [Head] be the shape of [Tail]? [MASK].
---
can [Head] be the shape of [Tail]? [MASK]!
does the [Head] have a shape of [Tail]? [MASK].
does the [Head] have a shape of [Tail]? [MASK]!
is [Head] of [Tail]? [MASK].
is [Head] of [Tail]? [MASK]!
Q: is [Head] of [Tail]? A: [MASK].
Question: is [Head] of [Tail]? Answer: [MASK].
[Tail] [Head]? [MASK].
is [Head] typically [Tail]? [MASK].
Material |  | can [Head] be made of [Tail]? [MASK]!
---
can [Head] be made of [Tail]? [MASK].
is [Head] made of [Tail]? [MASK]!
is [Head] made of [Tail]? [MASK].
is [Tail] the necessary material for making [Head]? [MASK].
is [Tail] the necessary material for making [Head]? [MASK]!
does [Head] consist of [Tail]? [MASK].
is [Head] made up of [Tail]? [MASK].
Q: is [Head] made of [Tail]? A: [MASK].
Question: is [Head] made of [Tail]? Answer: [MASK].
Table 11: Prompts for Causal Language Models Model | Task | Prompt
---|---|---
OPT | Size, Height, Temperature, Weight, Hardness |  | the [Head] is [Rel] than the [Tail].
---
[Head] is [Rel] than [Tail].
acutally, the [Head] is [Rel] than the [Tail].
acutally, [Head] is [Rel] than [Tail].
it is well-known that [Head] is [Rel] than [Tail].
[Head] is indeed [Rel] than [Tail].
the [Head] is indeed [Rel] than [Tail].
compared with the [Head], the [Tail] is [Rel].
a/(an) [Head] is [Rel] than a/(an) [Tail].
yes, [Head] is [Rel] than [Tail].
Color |  | [Head] can be of the color [Tail].
---
the [Head] can be of color [Tail].
the color of a(an) [Head] is [Tail].
the color of [Head] is [Tail].
the [Head] is in [Tail].
[Head] is [Tail].
what color is the [Head]? [Tail].
[Head]’s color is [Tail].
usually, [Head] is in [Tail].
[Head] is typically [Tail].
Shape |  | [Head] is usually [Tail].
---
what is the shape of [Head]? [Tail].
[Head] is typically [Tail].
[Head]’s shape is [Tail].
Material |  | [Head] is made of [Tail].
---
the [Head] is made of [Tail].
[Head] consists of [Tail].
the main material of [Head] is [Tail].
[Tail] is necessary material for making [Head].
the [Head] consists of [Tail].
the [Head] can be made of [Tail].
the [Head] is built with [Tail].
the [Head] contains [Tail].
the [Head] is made up of [Tail].
Table 12: Prompts used for VLMs of CLIP. Model | Task | Prompt
---|---|---
CLIP | All Tasks |  | a photo of a [Head]/[Attribute].
---
a photo of the [Head]/[Attribute].
a blurry photo of a [Head]/[Attribute].
a good photo of a [Head]/[Attribute].
a painting of a [Head]/[Attribute].
a bad photo of a [Head]/[Attribute].
a close-up photo of a [Head]/[Attribute].
a bright photo of the [Head]/[Attribute].
a photo of one [Head]/[Attribute].
a low resolution photo of a [Head]/[Attribute].