latexCommand menukeys

# An exploratory experiment on Hindi, Bengali hate-speech detection and
transfer learning using neural networks

Minh Tung Phung
Saarland University
<EMAIL_ADDRESS>
Jan Cloos
Saarland University
<EMAIL_ADDRESS>

###### Abstract

This work presents our approach to train a neural network to detect hate-
speech texts in Hindi and Bengali. We also explore how transfer learning can
be applied to learning these languages, given that they have the same origin
and thus, are similar to some extend. Even though the whole experiment was
conducted with low computational power, the obtained result is comparable to
the results of other, more expensive, models. Furthermore, since the training
data in use is relatively small and the two languages are almost entirely
unknown to us, this work can be generalized as an effort to demystify lost or
alien languages that no human is capable of understanding.

##  1 Introduction

The emergence of social networks such as Twitter and Facebook inarguably
brings up many benefits to humankind. Besides the pure joy of scrolling the
new-feeds list, these virtual environments provide people with the easiest way
of staying connected to others and also updated to real-time news and social
trends. This is especially true during the time of the historic COVID-19
pandemic, when social distancing is more important than ever and one of the
most convenient means for people to catch up is to post, comment, and reply on
social media. However, some individuals exploit the fact that online platforms
are mostly open and anonymous to insult and abuse other people who have
different interests, race, color, or religion, etc. A lot of research has been
conducted to automatically recognize toxic texts on social media platforms,
including work from Gaydhani et al. [5], which got an impressive accuracy of
95.6% on classifying English tweets into _Hateful_ , _Offensive_ , and _Clean_
, and from Badjatiya et al. [3] with an F1-score of 93% when categorizing
texts into _Racist_ , _Sexist_ , or _Neither_. While remarkable results were
obtained for English, much less work was directed to less popular languages
like Hindi. One of such attempts was the organization of the HASOC competition
in 2019 [12] in which one sub-task was to process a Hindi dataset (collected
from Twitter and Facebook) and differentiate _Hate-speech_ from _Non-
offensive_ content. We also make use of this dataset for our experiment.

One of the common architectures many participants chose was to fine-tune a
pre-trained BERT model (Devlin et al. [4]) using the dataset of interest,
alongside applying some additional transformations and processing (e.g.
Ranasinghe et al. [18], Mishra and Mishra [15]). This illustrates how transfer
learning has gone viral in the field of natural language processing (NLP). In
this paper, we discuss how learning can be transferred from Hindi to Bengali
and the benefits of this approach.

##  2 Data

We use the Hindi dataset from the Hate Speech and Offensive Content
Identification in Indo-European Languages competition (HASOC) 2019 [12]. The
training part consists of 4665 text sentences originated from Twitter and
Facebook. Each text has 3 labels corresponding to 3 sub-tasks of the
competition: the first label shows whether the text contains any type of
offensive content, the second label names the particular type of hate, and the
third label determines if the toxic content has specific targets or not. Note
that while the vocab size of the corpus is roughly 22K, the total number of
tokens is only about 135K, resulting in a low ratio of occurrences per token
on average. The test data follows a similar format but contains only 1318
texts. As the goal of the experiment is to distinguish hate and offensive
texts from normal contents, only the first label is used.

As for the Bengali dataset, we reuse the one from Romim et al. [19]. The
authors extracted comments from Facebook pages and Youtube videos whose topics
range from celebrity and sports to crime and politics. Each comment is marked
with a binary label indicating if it is hate or not. There are 30k labeled
comments in total. To alleviate transfer learning and easier comparison with
learning-from-scratch, we do random sampling on the Bengali dataset to get a
sample of equal size and label distribution as the Hindi dataset for both
training and test data. This sample Bengali data is used in the experiments to
be described in the next sections. Figure 1 compares the distributions and
statistics of these datasets.

Figure 1: Statistics of the datasets. For the Hindi and sample Bengali, only
the training data is considered. Tokenization is done by simply removing
punctuations and splitting by whitespace. Stopwords are included in the
statistics. a) The number of texts (sentences, comments). b) The average
number of tokens per text. c) The vocab size (the number of different tokens).
d) The average number of occurrences per distinct token. e) The proportion of
hate/offensive texts in the corpus.

##  3 Methodology

###  3.1 Overview

We break down our work into 3 phases: data pre-processing, training word-
embedding, and classification. During the first phase, raw texts are cleaned
and tokenized. In the second phase, we pre-train a word-embedding using a
word-2-vec model. These embeddings of words help the process of training the
text classifier in the third phase more smooth and achieve better performance.

Over the next sections, we present a total of 6 models: a baseline and an
optimized model for each of the 3 tasks: training a text classifier of Hindi
using Hindi data (Hindi-Hindi), training a text classifier of Bengali using
Bengali data (Bengali-Bengali), and training a text classifier of Bengali
using Bengali data together with knowledge transferred from the Hindi
classifier (Hindi-Bengali).

###  3.2 Baseline models

To create the baseline models for future comparison, we first clean the text
by removing the tagging of usernames (those starting with ’@’). Since both
datasets are originated from comments and posts on social media, a lot of
usernames can be found within the texts, however, it is clear that they do not
contribute to our objective of hate-speech detection. Punctuations are also
gotten rid of with the same argument. Furthermore, we notice that many texts
are partially composed of English words. We do not delete but lowercase them
for standardization. Stopwords are also dropped as a common practice. Note
that we keep the hashtags since they can potentially contain information to
link different texts to each other and even more directly, some hashtags might
involve explicitly offensive content.

After the cleaning phase, word embedding is trained with the Skip-gram
architecture as proposed by Mikolov et al. [13]. About the choices of
hyperparameters, we use an embedding size of 300 and window size of 10, as
suggested in the paper and the corresponding reference code [1]. However, even
though the authors originally used Adagrad as the optimizer, we replace it
with Adam since Adagrad has a drawback about its cumulative penalty that makes
the model unable to learn after a certain number of training steps. Adam, on
the other hand, has been the default optimizer of most neural networks for
several years. The loss function is cross-entropy (the CrossEntropyLoss on
Pytorch, which is equivalent to NLLLoss applied on LogSoftmax of the output,
so this loss function complies with the original paper).

The text classifier is a simple neural network with an embedding layer at the
bottom (weights of this layer were learned from the previous phase), a multi-
head attention layer (MHA) with layer normalization in the middle, and a fully
connected at the top to give predictions. There is also a shortcut connection
from the embedding layer to the output of the MHA to facilitate training. The
binary cross-entropy with logit loss and Adam are chosen as the loss function
and the optimizer by default, respectively. Figure 2 (left) outlines the three
phases of the baseline model.

Figure 2: Model architectures: the baseline model pipeline for Hindi-Hindi and
Bengali-Bengali text classification is shown on the left, the enhanced model
pipeline is shown on the right.

###  3.3 Enhanced models

To improve model performance, we do a number of experiments to better refine
data, learn a more meaningful word embedding, and find more suitable network
structures.

We spot that aside from usernames, the text also contains web URLs, which need
to be removed. Moreover, there are numbers in the text. Treating each number
as a token is probably not effective since most of the numbers only appear
once in the corpus, making it hard to learn their suitable embedding.
Moreover, numbers should not be entirely deleted as they may enclose useful
information. To cope with this, we keep the occurrences of numbers 0, 1, and 2
(since they appear more times and may have specific meaning) but replace all
others with some special tokens corresponding to the number of digits. That
is, for example, all numbers with 2 digits are replaced with a special token,
the numbers with 3 digits are replaced with another special token. This simple
replacement (i.e. binning) groups similar numbers into the same bin, whose
effect is to increase the ratio of occurrences per token, which in turn
facilitates learning. We also try forming 2-gram phrases using the statistical
approach as described by Mikolov et al. [14]. However, this even reduces model
performance. Another attempt was to form 2-grams using only the absolute
counts (i.e. without dividing by the counts of 2 individual unigrams), which
gives no better results. We suspect the reason is that the small dataset size
leads to a non-reliable statistical inference. We will see in the following
that other techniques that depend on data statistics also give unfavorable
effects on this dataset. One of those attempts was to try using Byte-Pair
Encoding (BPE), which was introduced by Sennrich et al. [20], with the help of
the sentencepiece library by Kudo [10]. BPE is a tokenization strategy based
on the frequency of adjacent bytes. First, the vocabulary is initialized as
the set of all byte-codes in the training corpus. Then, it iteratively
computes the most frequent co-occurrences of words to form new symbols to
enlarge the vocabulary. This process terminates when the vocabulary size
reaches the desired size, which is a hyperparameter the user defined from the
start. The BPE tokenizer has no assumptions about the language (some
assumptions of other tokenizers are, for example, that words are separated by
space, and that dots are used to end a sentence), which makes it seemingly
ideal for dealing with entirely unknown languages. However, in our case, using
BPE instead of simple splitting by whitespace decreases validation accuracy by
about 3 to 4%.

For the training of word-embedding, we choose the Skip-gram with Negative
Sampling, which was proposed by Mikolov et al. [14]. According to the authors,
this variant of Skip-gram not only gets rid of the expensive cost of the
softmax layer in the original Skip-gram model but also achieves very good
results in terms of performance. This is achieved by altering the network’s
objective from predicting nearby words (context) given the input words
(center) to assessing if each pair of input words are close to each other. We
use the window size value of 5 after trying out a few options. The ratio of
negative samples per positive data point is chosen to be 10, this also matches
the suggestion in the paper, saying that for small datasets, the value should
be in the range 5-20 [14]. For noise distribution, we exploit the recommended
Unigram distribution raised to the power of $\frac{3}{4}$ (this indeed
increases the accuracy of the validation set). Subsampling of frequent words
is also kept from the basic models. Moreover, we employ the Xavier
initialization for both center and context embedding matrices. The Xavier
initialization, introduced by Glorot and Bengio [6], allows the networks with
tanh or sigmoid activation functions to keep the same variance across layers
during training, resulting in not only faster but also sometimes better
convergence [6, 16]. As we use sigmoid activation in the current network, we
try out this initialization method and find that the validation accuracy is
increased by about 1 to 3% for all models. Another alternative to Skip-gram is
GloVe [17]. However, as we test this architecture on the Hindi dataset, the
classifier’s performance decreases by roughly 3 to 4% in terms of accuracy.
Interestingly, Badjatiya et al. [3] observed the same negative impact of GloVe
on the same task (hate-speech detection) but with a different dataset (which
has 16K texts, more than our Hindi dataset but is still quite small). Since
GloVe is based on the statistic matrix of global counts over all word co-
occurrences, we suppose it is suffering from the same problem we encountered
with BPE as described above. Lastly, we apply adaptive learning rate with an
exponential decay to balance between fast training speed and good convergence.

About the text classifier, we examine 2 main architectures: Long-Short Term
Memory (LSTM) [7] and the Encoder (from the Transformer by Vaswani et al.
[22]). For LSTM, both the normal (unidirectional) and bidirectional LSTM are
studied, with the bidirectional version slightly outperforming the other one.
For LSTM models, we experiment with doubling the inputs (e.g. if the original
text is "this dish is delicious", we input to the model the doubled text "this
dish is delicious this dish is delicious"), inspired by the fact that it is
often easier for humans to understand a difficult sentence (or paragraph) by
re-reading it one more time. However, the observed changes in accuracy are
minimal. We next explore the Encoder architecture and find that both training
speed and prediction performance are improved by a large margin. Since the
attention mechanism inside the Encoder allows parallel computation, it is
expected that the learning progress is faster compared to the sequential
process of LSTM. Furthermore, the better performance is also not of a surprise
since the Transformers have taken over Recurrent neural networks for the
state-of-the-art performance in many different tasks in recent years (e.g. as
shown in the work by Lakew et al. [11] and Karita et al. [9]).

For the choice of normalization, even though Batch Normalization (Ioffe and
Szegedy [8]) is very popular, it is not straight-forward how it should be
applied on text input, where the text lengths (number of tokens) might vary.
Instead, we try out Layer Normalization (Lei Ba et al. [2]) and Instance
Normalization (Ulyanov et al. [21]) and find the former one works better and
is more stable. This should be due to the fact that Layer Normalization often
gives more regularization effects since it is computed over a wider range of
the input data. To even enforce more regularization, Dropout is applied after
every multi-head attention layer with zeroing-probability of $0.7$. Moreover,
we try augmenting the input texts by randomly removing some small portions of
tokens and/or replacing them with similar tokens (similarity is computed based
on the Cosine-similarity of word-embeddings), but no improvement is recorded.
On the other hand, the Sine-Cosine Positional Encoding (as described by
Vaswani et al. [22]) is working well. Since the attention mechanism does not
take in the tokens sequentially (as in the case of LSTMs), we need the
Positional Encoding to include the ordering information of tokens into the
inputs. By adding this encoding to the word-embedding right before feeding it
to the first Encoder block, we observe that the maximum validation accuracy
rises slightly, and even more importantly, it fluctuates around the highest
point as the training loss approaches zero (without Positional Encoding, the
validation accuracy may go down by as much as 4-5% from the highest point as
the number of training epochs increases).

For the Bengali dataset, we take data (excluding the testing set) for learning
word-embedding. During the training of the classifier, only the sample
training set (which is equal in size to the Hindi training data) is used.

In the end, we settle down with the overall pipeline as shown in Figure 2
(right).

Figure 3: Test accuracy for 3 baseline and 3 enhanced models. No layers are
fixed during the last tuning/training phase.

###  3.4 Transfer learning

Since Hindi and Bengali have the same origins in Sanskrit and are both widely
spoken in the same region (India), we expect them to share some similarities
in grammatical syntax, which motivates transfer learning. In particular, we
experiment with reusing the non-embedding layers of the Hindi-Hindi classifier
for the Hindi-Bengali classifier. Put differently, the Hindi-Bengali
classifier is formed by replacing the word-embedding layer of the Hindi-Hindi
classifier with another, untrained word-embedding layer of the Bengali
language. During the training of the Hindi-Bengali classifier, either only the
word-embedding layer learns (if we fix all the transferred layers) or all
layers learn together (if we do not fix any layers, in this case, the
transferred weights act as a weight initialization). We explore both cases to
see their effects.

With the help of knowledge transfer, we expect the convergent rate of the
Hindi-Bengali classifier would be faster than learning from scratch, while
still maintain reasonable performance. Furthermore, we hypothesize that if we
fix all the transferred weights (i.e. the non-embedding layers), the resulting
Bengali word-embedding may likely be forced to converge to an equivalent
representation to the Hindi word-embedding of the Hindi-Hindi classifier. In
other words, after training the Hindi-Bengali classifier, we may be able to
obtain a rough translation between Hindi and Bengali languages using the two
word-embedding matrices.

##  4 Results

Figure 4: The Macro-f1 score of the enhanced Hindi-Hindi model, taking over 5
different runs.

To tune the hyperparameters, we first split the data into a big part (80%) and
a small part (20%). The models are then trained on 80% of the data and the
hyperparameters are adjusted based on their performance on the left-out 20%.
When hyperparameter tuning is done, we retrain the models once again on all
training data and evaluate them using the test set. The results are reported
based on taking the average of 5 runs.

First, we examine how the models behave if no layers are fixed during the
training of the classifier. In other words, we use transfer learning as a
weight initialization method. The result is shown in Figure 3 We observe that
all enhanced models outperform their baseline counterparts. The biggest rise
comes from the Hindi-Hindi models, in which a lift of about 10% in accuracy is
recorded. Moreover, while the baseline classifier tends to get worse after
reaching the highest accuracy after epoch 8, the enhanced version does not
suffer from the same problem. Regarding the Bengali-Bengali models, the
average gain is around 4%, noticing the fact that the baseline’s performance
is already high (the average accuracy from epoch 10 onward is 78.2%). The
knowledge-transferred models, Hindi-Bengali, show a slight increment of about
2%.

It is interesting to see that the performance-decaying effect only happens to
the (baseline) Hindi-Hindi but not the Bengali-Bengali model, even though the
two datasets share the same size and label distribution. The source of the
problem may lie in the differences in vocabulary size, the average text size,
the number of occurrences per distinct token, or the intrinsic properties of
these languages themselves. This remains an unanswered question. Nevertheless,
none of the more-complicated, enhanced models suffers from the same issue.

Table 1: Best accuracy over all tasks | Hindi-Hindi | Hindi-Bengali | Bengali-Bengali
---|---|---|---
Config | enhanced, no-fix | enhanced, fix | enhanced, fix
|  | non-embedding | embedding
Accuracy | 80.73% | 80.40% | 82.98%
Figure 5: Test accuracy for 3 baseline and 3 enhanced models. The embedding
layer of the Hindi-Hindi and Bengali-Bengali models, and non-embedding layers
of the Hindi-Bengali models are kept unchanged during the training of the
classifiers.

Figure 4 gives more insight into the performance of the enhanced Hindi-Hindi
model. It shows the Macro-f1 score of all 5 component runs together with the
average. After 15 epochs, the average score reaches 0.8057. This result, if to
be compared with the teams competed in the HASOC 2019 competition [12], can be
well placed in the top 10.

Next, we inspect the models in case the transferred knowledge (for Hindi-Hindi
and Bengali-Bengali models, the embedding weights are transferred from the
Skip-grams, and for Hindi-Bengali models, the non-embedding weights are reused
from the Hindi-Hindi classifiers) is fixed during the training of the
classifiers. This might be a more commonly-used practice of transfer learning
in general. The result is shown in Figure 5. While the other 2 models still
exhibit significant advancement over their baselines, there is no big change
in the accuracy of the enhanced Hindi-Bengali model. In fact, this is mostly
caused by the surprisingly good performance of the Hindi-Bengali baseline, it
gives approximately 80% accuracy on the test set from epoch 6 onward.

The best test accuracy over all configurations are shown in Table 1.

Lastly, we qualitatively examine if the word-to-word translation can be
exploited as a side effect of transfer learning. We pick 10 common-sense words
in English. For each of these, we get its equivalent translations in Hindi and
Bengali using Google Translate, and compute the Cosine-similarity ranking of
the 2 embedding vectors in comparison to other words in the 2 vocabularies. We
use the embedding weights of the enhanced Hindi-Hindi and Hindi-Bengali models
for this test. The Hindi-Bengali classifier is trained with all non-embedding
layers transferred (from the Hindi-Hindi classifier) and fixed. Figure 6 shows
the result. Unfortunately, no sight of meaningful translation can be observed.
We hypothesize that the Bengali embedding layer has failed to converge to an
equivalent representation as of the Hindi embedding. This might be due to the
small datasets, sub-optimal optimizations, and/or the differences between the
2 languages. More work needs to be conducted to investigate this issue.

Figure 6: Hindi and Bengali word-to-word translation using Cosine-similarity
ranking.
A ranking of 1 means perfect translation.

##  5 Conclusion

In this paper, we explore how to build a hate-speech detection model pipeline
either from scratch or through transfer learning. Despite the simple
architecture and low computational cost, the models give comparable
performance to more complex models like BERT. With transfer learning, we show
that by reusing the Hindi model and tuning only the word-embedding layer, we
can obtain a Bengali model with marginal difference in performance with a
model built from scratch. This shows a sign of simpler transfer learning
compared to common practice when we often need to tune several fully connected
layers of the pre-trained models to match our tasks. We also make the first
attempt towards deriving a word-to-word translator from transfer learning.
This could serve as a good direction for future research.

## References

  * [1] G. C. Archive.  Word2vec (2013).  https://code.google.com/archive/p/word2vec.  Accessed: 2021-03-26.
  * [2] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [3] P. Badjatiya, S. Gupta, M. Gupta, and V. Varma.  Deep learning for hate speech detection in tweets.  In Proceedings of the 26th international conference on World Wide Web companion, pages 759–760, 2017.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] A. Gaydhani, V. Doma, S. Kendre, and L. Bhagwat.  Detecting hate speech and offensive language on twitter using machine learning: An n-gram and tfidf based approach.  arXiv preprint arXiv:1809.08651, 2018.
  * [6] X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [7] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [8] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [9] S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang, et al.  A comparative study on transformer vs rnn in speech applications.  In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 449–456. IEEE, 2019.
  * [10] T. Kudo.  Sentencepiece (2018).  https://github.com/google/sentencepiece.  Accessed: 2021-03-26.
  * [11] S. M. Lakew, M. Cettolo, and M. Federico.  A comparison of transformer and recurrent neural networks on multilingual neural machine translation.  arXiv preprint arXiv:1806.06957, 2018.
  * [12] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, and A. Patel.  Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages.  In Proceedings of the 11th forum for information retrieval evaluation, pages 14–17, 2019.
  * [13] T. Mikolov, K. Chen, G. Corrado, and J. Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [14] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  arXiv preprint arXiv:1310.4546, 2013.
  * [15] S. Mishra and S. Mishra.  3idiots at hasoc 2019: Fine-tuning transformer neural networks for hate speech identification in indo-european languages.  In FIRE (Working Notes), pages 208–213, 2019.
  * [16] A. Ng and K. Katanforoosh.  Xavier initialization and regularization - course lecture cs230.  https://cs230.stanford.edu/section/4/.  Accessed: 2021-03-26.
  * [17] J. Pennington, R. Socher, and C. D. Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [18] T. Ranasinghe, M. Zampieri, and H. Hettiarachchi.  Brums at hasoc 2019: Deep learning models for multilingual hate speech and offensive language identification.  In FIRE (Working Notes), pages 199–207, 2019.
  * [19] N. Romim, M. Ahmed, H. Talukder, and M. S. Islam.  Hate speech detection in the bengali language: A dataset and its baseline evaluation.  arXiv preprint arXiv:2012.09686, 2020.
  * [20] R. Sennrich, B. Haddow, and A. Birch.  Neural machine translation of rare words with subword units.  arXiv preprint arXiv:1508.07909, 2015.
  * [21] D. Ulyanov, A. Vedaldi, and V. Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  arXiv preprint arXiv:1607.08022, 2016.
  * [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.


# PIGLeT:
Language Grounding Through Neuro-Symbolic Interaction in a 3D World

Rowan Zellers♠ Ari Holtzman♠ Matthew Peters♡
Roozbeh Mottaghi♡ Aniruddha Kembhavi♡ Ali Farhadi♠ Yejin Choi♠♡
♠Paul G. Allen School of Computer Science & Engineering, University of
Washington
♡Allen Institute for Artificial Intelligence
https://rowanzellers.com/piglet

###### Abstract

We propose PIGLeT: a model that learns physical commonsense knowledge through
interaction, and then uses this knowledge to ground language. We factorize
PIGLeT into a physical dynamics model, and a separate language model. Our
dynamics model learns not just what objects _are_ but also what they _do_ :
glass cups break when thrown, plastic ones don’t. We then use it as the
interface to our language model, giving us a unified model of linguistic form
and grounded meaning. PIGLeT can read a sentence, simulate neurally what might
happen next, and then communicate that result through a literal symbolic
representation, or natural language.

Experimental results show that our model effectively learns world dynamics,
along with how to communicate them. It is able to correctly forecast “what
happens next” given an English sentence over 80% of the time, outperforming a
100x larger, text-to-text approach by over 10%. Likewise, its natural language
summaries of physical interactions are also judged by humans as more accurate
than LM alternatives. We present comprehensive analysis showing room for
future work.

##  1 Introduction

As humans, our use of language is linked to the physical world. To process a
sentence like “the robot turns on the stove, with a pan on it” (Figure 1) we
might imagine a physical Pan object. This meaning representation in our heads
can be seen as a part of our commonsense world knowledge, about what a Pan is
and does. We might reasonably predict that the Pan will become Hot – and if
there’s an Egg on it, it would become cooked.

Figure 1: PIGLeT. Through physical interaction in a 3D world, we learn a model
for what actions do to objects. We use our physical model as an interface for
a language model, jointly modeling elements of language _form_ and _meaning_.
Given an action expressed symbolically or in English, PIGLeT can simulate what
might happen next, expressing it symbolically or in English.

As humans, we learn such a commonsense world model through interaction. Young
children learn to reason physically about basic objects by manipulating them:
observing the properties they have, and how they change if an action is
applied on them (Smith and Gasser, 2005). This process is hypothesized to be
crucial to how children learn language: the names of these elementary objects
become their first “real words” upon which other language is scaffolded (Yu
and Smith, 2012).

Figure 2: PIGPeN, a setting for few-shot language-world grounding. We collect
data for 280k physical interactions in THOR, a 3D simulator with 20 actions
and 125 object types, each with 42 attributes (e.g. isBroken). We annotate 2k
interactions with English sentences describing the initial world state, the
action, and the action result.

In contrast, the dominant paradigm today is to train large language or vision
models on _static data_ , such as language and photos from the web. Yet such a
setting is fundamentally limiting, as suggested empirically by psychologists’
failed attempts to get kittens to learn passively (Held and Hein, 1963). More
recently, though large Transformers have made initial progress on benchmarks,
they also have frequently revealed biases in those same datasets, suggesting
they might not be solving underlying tasks (Zellers et al., 2019b). This has
been argued philosophically by a flurry of recent work arguing that no amount
of language _form_ could ever specify language _meaning_ (McClelland et al.,
2019; Bender and Koller, 2020; Bisk et al., 2020); connecting back to the
Symbol Grounding Problem of Harnad (1990).

In this paper, we investigate an alternate strategy for learning physical
commonsense through interaction, and then transferring that into language. We
introduce a model named PIGLeT, short for Physical Interaction as Grounding
for Language Transformers. We factorize an embodied agent into an explicit
model of world dynamics, and a model of language form. We learn the dynamics
model through _interaction_. Given an action heatUp applied to the Pan in
Figure 1, the model learns that the Egg on the pan becomes Hot and Cooked, and
that other attributes do not change.

We integrate our dynamics model with a pretrained language model, giving us a
joint model of linguistic _form_ and _meaning_. The combined PIGLeT can then
reason about the physical dynamics implied by English sentences describing
actions, predicting literally what might happen next. It can then communicate
that result either symbolically or through natural language, generating a
sentence like ‘The egg becomes hot and cooked.” Our separation between
physical dynamics and language allows the model to learn about physical
commonsense from the physical world itself, while also avoiding recurring
problems of artifacts and biases that arise when we try to model physical
world understanding solely through language.

We study this through a new environment and evaluation setup called PIGPeN,
short for Physical Interaction Grounding Paired with Natural Language. In
PIGPeN, a model is given unlimited access to an environment for pretraining,
but only 500 examples with paired English annotations. Models in our setup
must additionally generalize to novel ‘unseen’ objects for which we
intentionally do not provide paired language-environment supervision. We build
this on top of the THOR environment Kolve et al. (2017), a physics engine that
enables agents to perform contextual interactions (Fig 2) on everyday objects.

Experiments confirm that PIGLeT performs well at grounding language with
meaning. Given a sentence describing an action, our model predicts the
resulting object states correctly over 80% of the time, outperforming even a
100x larger model (T5-11B) by over 10%. Likewise, its generated natural
language is rated by humans as being more correct than equivalently-sized
language models. Last, it can generalize in a ‘zero-shot’ way to objects that
it has never read about before in language.

In summary, we make three key contributions. First, we introduce PIGLeT, a
model decoupling physical and linguistic reasoning. Second, we introduce
PIGPeN, to learn and evaluate the transfer of physical knowledge to the world
of language. Third, we perform experiments and analysis suggesting promising
avenues for future work.

##  2 PIGPeN: A Resource for Neuro-Symbolic Language Grounding

We introduce PIGPeN as a setting for learning and evaluating physically
grounded language understanding. An overview is shown in Figure 2. The idea is
that an agent gets access to an interactive 3D environment, where it can learn
about the world through interaction – for example, that objects such as a Vase
can become Broken if thrown. The goal for a model is to learn natural language
_meaning_ grounded in these interactions.

Task definition. Through interaction, an agent observes the interplay between
objects $\bm{o}\in\mathcal{O}$ (represented by their attributes) and actions
$\bm{a}\in\mathcal{A}$ through the following transition:

$\underbrace{\\{\bm{o}_{1},\ldots,\bm{o}_{N}\\}}_{\vec{\bm{o}},\textrm{ state
pre-
action}}\times\bm{a}\rightarrow\underbrace{\\{\bm{o}^{\prime}_{1},\ldots,\bm{o}^{\prime}_{N}\\}}_{\vec{\bm{o}}^{\prime},\textrm{
state post-action}}.$ (1)

Actions change the state of a subset of objects: turning on a Faucet affects a
nearby Sink, but it will not change a Mirror on the wall.

To encourage learning from interaction, and not just language, an agent is
given a small number of natural language annotations of transitions. We denote
these sentences as $\bm{s}_{\vec{\bm{o}}}$, describing the state pre-action,
$\bm{s}_{\bm{a}}$ the action, and $\bm{s}_{\vec{\bm{o}}^{\prime}}$ the state
post-action respectively. During evaluation, an agent will sometimes encounter
new objects $\bm{o}$ that were not part of the paired training data.

We evaluate the model’s transfer in two ways:

  1. a.

PIGPeN-NLU. A model is given object states $\vec{\bm{o}}$, and an English
sentence $\bm{s}_{\bm{a}}$ describing an action. It must predict the grounded
object states $\vec{\bm{o}}^{\prime}$ that result after the action is taken.

  2. b.

PIGPeN-NLG. A model is given object states $\vec{\bm{o}}$ and a literal action
$\bm{a}$. It must generate a sentence $\bm{s}_{\vec{\bm{o}}^{\prime}}$
describing the state post-action.

We next describe our environment, feature representation, and language
annotation process.

###  2.1 Environment: THOR

We use AI2-THOR as an environment for this task Kolve et al. (2017). In THOR,
a robotic agent can navigate around and perform rich contextual interactions
with objects in a house. For instance, it can grab an Apple, slice it, put it
in a Fridge, drop it, and so on. The state of the Apple, such as whether it is
sliced or cold, changes accordingly; this is not possible in many other
environments.

In this work, we use the underlying THOR simulator as a proxy for grounded
meaning. Within THOR, it can be seen as a ‘complete’ meaning representation
Artzi et al. (2013), as it fully specifies the kind of grounding a model can
expect in its perception within THOR.

Objects. The underlying THOR representation of each object $\bm{o}$ is in
terms of 42 attributes; we provide a list in Appendix B. We treat these
attributes as words specific to an attribute-level dictionary; for example,
the temperature Hot is one of three possible values for an object’s
temperature; the others being Cold and RoomTemp.

Actions. An action $\bm{a}$ in THOR is a function that takes up to two objects
as arguments. Actions are highly contextual, affecting not only the arguments
but potentially other objects in the scene (Figure 2). We also treat action
names as words in a dictionary.

Filtering out background objects. Most actions change the state of only a few
objects, yet there can be many objects in a scene. We keep annotation and
computation tractable by having models predict (and humans annotate) possible
changes of at most two key objects in the scene. As knowing when an object
_doesn’t_ change is also important, we include non-changing objects if fewer
than two change.

Exploration. Any way of exploring the environment is valid for our task,
however, we found that exploring _intentionally_ was needed to yield good
coverage of interesting states. Similar to prior work for instruction
following Shridhar et al. (2020), we designed an oracle to collect diverse and
interesting trajectories $\\{\vec{\bm{o}},\bm{a},\vec{\bm{o}}^{\prime}\\}$.
Our oracle randomly selects one of ten high level tasks, see Appendix B for
the list. These in turn require randomly choosing objects in the scene; e.g. a
Vase and a Laptop in Figure 2. We randomize the manner in which the oracle
performs the task to discover diverse situations.

In total, we sampled 20k trajectories. From these we extracted 280k
transitions (Eqn 1’s) where at least one object changes state, for training.

###  2.2 Annotating Interactions with Language

####  2.2.1 Data Selection for Annotation

We select 2k action state-changes from trajectories held out from the training
set. We select them while also balancing the distribution of action types to
ensure broad coverage in the final dataset. We are also interested in a
model’s ability to generalize to new object categories – beyond what it has
read about, or observed in a training set. We thus select 30 objects to be
“unseen,” and exclude these from paired environment-language training data. We
sample 500 state transitions, containing only “seen” objects to be the
training set; we use 500 for validation and 1000 for testing.

####  2.2.2 Natural Language Annotation

Workers on Mechanical Turk were shown an environment in THOR _before_ and
_after_ a given action $\bm{a}$. Each view contains the THOR attributes of the
two key objects. Workers then wrote three English sentences, corresponding to
$\bm{s}_{\vec{\bm{o}}}$, $\bm{s}_{\bm{a}}$, and
$\bm{s}_{\vec{\bm{o}}^{\prime}}$ respectively. Workers were instructed to
write at a particular level of detail: enough so that a reader could infer
“what happens next” from $\bm{s}_{\vec{\bm{o}}}$ and $\bm{s}_{\bm{a}}$, yet
without mentioning redundant attributes.We provide more details in Appendix 6.

Figure 3: PIGLeT architecture. We pretrain a model of physical world dynamics
by learning to transform objects $\vec{\bm{o}}$ and actions $\bm{a}$ into new
updated objects $\vec{\bm{o}}^{\prime}$. Our underlying world dynamics model –
the encoder, the decoder, and the action application module, can augment a
language model with grounded commonsense knowledge.

##  3 Modeling PIGLeT

In this section, we describe our PIGLeT model. First, we learn a neural
physical dynamics model from interactions, and second, integrate with a
pretrained model of language form.

###  3.1 Modeling Physical Dynamics

We take a neural, auto-encoder style approach to model world dynamics. An
object $\bm{o}$ gets encoded as a vector
$\mathbf{h}_{\bm{o}}\in\mathbb{R}^{d_{\textrm{o}}}$. The model likewise
encodes an action $\bm{a}$ as a vector
$\mathbf{h}_{\bm{a}}\in\mathbb{R}^{d_{\textrm{a}}}$, using it to manipulate
the hidden states of all objects. The model can then decode any object hidden
representation back into a symbolic form.

####  3.1.1 Object Encoder and Decoder

We use a Transformer Vaswani et al. (2017) to encode objects into vectors
$\mathbf{o}\in\mathbb{R}^{d_{\textrm{o}}}$, and then another to decode from
this representation.

Encoder. Objects $\bm{o}$ are provided to the encoder as a set of attributes,
with categories
$c_{1}\mathinner{\kern-1.99997pt,\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt,}c_{n}$.
Each attribute $c$ has its own vocabulary and embedding $\mathbf{E}_{c}$. For
each object $\bm{o}$, we first embed all the attributes separately and feed
the result into a Transformer encoder $T_{enc}$. This gives us (with position
embeddings omitted for clarity):

$\mathbf{h}_{\bm{o}}=\mathbf{T}_{\textrm{enc}}\Big{(}\mathbf{E}_{1}(o_{1}),\dotsc,\mathbf{E}_{c_{n}}(o_{c_{n}})\Big{)}$
(2)

Decoder. We can then convert back into the original symbolic representation
through a left-to-right Transformer decoder, which predicts attributes one-by-
one from $c_{1}$ to $c_{n}$. This captures the inherent correlation between
attributes, while making no independence assumptions, we discuss our ordering
in Appendix A.2. The probability of predicting the next attribute
$o_{c_{i+1}}$ is then given by:

$\kern-8.00003ptp(o_{c_{i+1}}\kern-1.00006pt|\mathbf{h}_{\bm{o}},\bm{o}_{:c_{i}}\kern-1.00006pt){=}\mathbf{T}_{\textrm{dec}}\kern-1.00006pt\Big{(}\kern-1.00006pt\mathbf{h}_{\bm{o}},\kern-1.49994pt\mathbf{E}_{1}\kern-1.99997pt(o_{1}\kern-1.00006pt)\mathinner{\kern-1.99997pt,\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt,}\mathbf{E}_{c_{i}}\kern-1.00006pt(o_{c_{i}}\kern-1.00006pt)\kern-1.00006pt\Big{)}\kern-5.0pt$
(3)

####  3.1.2 Modeling actions as functions

We treat actions $\bm{a}$ as functions that transform the state of all objects
in the scene. Actions in our environment take at most two arguments, so we
embed the action $\bm{a}$ and the names of its arguments, concatenate them,
and pass the result through a multilayer perceptron; yielding a vector
representation $\mathbf{h}_{\bm{a}}$.

Applying Actions. We use the encoded action $\mathbf{h}_{\bm{a}}$ to transform
all objects in the scene, obtaining updated representations
$\hat{\mathbf{h}}_{\bm{o}^{\prime}}$ for each one. We take a _global_
approach, jointly transforming all objects. This takes into account that
interactions are contextual: turning on a Faucet might fill up a Cup if and
only if there is one beneath it.

Letting the observed objects in the interaction be $\bm{o}_{1}$ and
$\bm{o}_{2}$, with encodings $\mathbf{h}_{\bm{o_{1}}}$ and
$\mathbf{h}_{\bm{o_{2}}}$ respectively, we model the transformation via the
following multilayer perceptron:

$[\hat{\mathbf{h}}_{\bm{o_{1}}^{\prime}},\hat{\mathbf{h}}_{\bm{o_{2}}^{\prime}}]=\textrm{MLP}_{\textrm{apply}}\Big{(}\big{[}\mathbf{h}_{\bm{a}},\mathbf{h}_{\bm{o_{1}}},\mathbf{h}_{\bm{o}_{2}}\big{]}\Big{)}.$
(4)

The result can be decoded into symbolic form using the object decoder
(Equation 3).

####  3.1.3 Loss function and training

We train our dynamics model on
($\vec{\bm{o}}$,$\bm{a}$,$\vec{\bm{o}}^{\prime}$) transitions. The model
primarily learns by running $\vec{\bm{o}}$,$\bm{a}$ through the model,
predicting the updated output state $\hat{\mathbf{h}}_{\bm{o}^{\prime}}$, and
minimizing the cross-entropy of generating attributes of the real changed
object $\vec{\bm{o}}^{\prime}$. We also regularize the model by encoding
objects $\vec{\bm{o}},\vec{\bm{o}}^{\prime}$ and having the model learn to
reconstruct them. We weight all these cross-entropy losses equally. We discuss
our architecture in Appendix A.1; it uses 3-layer Transformers, totalling 17M
parameters.

###  3.2 Language Grounding

After pretraining our physical dynamics model, we integrate it with a
Transformer Language Model (LM). In our framework, the role of the LM will be
to both encode natural language sentences of actions into a hidden state
approximating $\mathbf{h}_{\bm{a}}$, as well as summarizing the result of an
interaction ($\vec{\bm{o}}$,$\bm{a}$,$\vec{\bm{o}}^{\prime}$) in natural
language.

Choice of LM. Our framework is compatible with any language model. However, to
explore the impact of pretraining data on grounding later in this paper, we
pretrain our own with an identical architecture to the smallest GPT2 (Radford
et al. (2019); 117M). To handle both classification and generation well, we
mask only part of the attention weights out, allowing the model to encode a
“prefix” bidirectionally; it generates subsequent tokens left-to-right Dong et
al. (2019). We pretrain the model on Wikipedia and books; details in Appendix
D.

We next discuss architectural details of performing the language transfer,
along with optimization.

####  3.2.1 Transfer Architecture

English actions to vector form. Given a natural language description
$\bm{s}_{\bm{a}}$ of an action $\bm{a}$, like “The robot throws the vase,” for
PIGPeN-NLU, our model will learn to parse this sentence into a neural
representation $\mathbf{h}_{\bm{a}}$, so the dynamics model can simulate the
result. We do this by encoding $\bm{s}_{\bm{a}}$ through our language model,
$\mathbf{T}_{LM}$, with a learned linear transformation over the resulting
(bidirectional) encoding. The resulting vector $\mathbf{h}_{\bm{s}_{\bm{a}}}$
can then be used by Equation 4.

Summarizing the result of an action. For PIGPeN-NLG, our model simulates the
result of an action $\bm{a}$ neurally, resulting in a predicted hidden state
$\hat{\mathbf{h}}_{\bm{o}}$ for each object in the scene $\bm{o}$. To write an
English summary describing “what changed,” we first learn a lightweight fused
representation of the transition, aggregating the initial and final states,
along with the action, through a multilayer perceptron. For each object
$\bm{o}_{i}$ we have:

$\mathbf{h}_{\Delta\bm{o}_{i}}=\textrm{MLP}_{\Delta}([\mathbf{h}_{\bm{o}_{i}},\hat{\mathbf{h}}_{\bm{o_{i}}^{\prime}},\mathbf{h}_{\bm{a}}]).$
(5)

We then use the sequence
$[\mathbf{h}_{\Delta\bm{o}_{1}},\mathbf{h}_{\Delta\bm{o}_{2}}]$ as
bidirectional context for our our LM to decode from. Additionally, since our
test set includes novel objects not seen in training, we provide the names of
the objects as additional context for the LM generator (e.g. ‘Vase, Laptop’);
this allows a LM to copy those names over rather than hallucinate wrong ones.
Importantly we only provide the surface-form names, not underlying information
about these objects or their usage as with few-shot scenarios in the recent
GPT-3 experiments Brown et al. (2020) – necessitating that PIGLeT learns what
these names _mean_ through interaction.

####  3.2.2 Loss functions and training.

Modeling text generation allows us to incorporate a new loss function, that of
minimizing the log-likelihood of generating each
$\bm{s}_{\vec{\bm{o}}^{\prime}}$ given previous words and the result of
Equation 5:

$p(s^{\textrm{post}}_{i+1}|\bm{s}_{\vec{\bm{o}^{\prime}},1:i})=\mathbf{T}_{\textrm{LM}}(\mathbf{h}_{\Delta\bm{o}_{1}},\mathbf{h}_{\Delta\bm{o}_{2}},\bm{s}_{\vec{\bm{o}^{\prime}},1:i}).$
(6)

We do the same for the object states $\bm{s}_{\vec{\bm{o}}}$ pre-action, using
$\mathbf{h}_{\bm{o}_{i}}$ as the corresponding hidden states.

For PIGPeN-NLU, where no generation is needed, optimizing Equation 5 is not
strictly necessary. However, as we will show later, it helps provide
additional signal to the model, improving overall accuracy by several
percentage points.

##  4 Experiments

|  Model | Accuracy (%)
---|---|---
| Val | Test
| Overall | Seen | Unseen
|  No Change | 27.4 | 25.5 | 29.9 | 24.0
text-to-text |  GPT3-175B Brown et al. (2020) | 23.8 | 22.4 | 22.4 | 21.4
T5-11B Raffel et al. (2019) | 68.5 | 64.2 | 79.5 | 59.1
T5-3B | 66.6 | 63.3 | 77.1 | 58.7
T5-Large | 56.5 | 54.1 | 69.2 | 49.1
T5-Base | 56.0 | 53.9 | 69.2 | 48.8
T5-Small | 39.9 | 36.2 | 57.0 | 38.0
BERT style |  Alberti et al.2019, Pretrained Dynamics | 61.3 | 53.9 | 71.4 | 48.1
Alberti et al.2019 |  09.7 |  06.8 | 16.2 |  03.7
G&D2019, Pretrained Dynamics | 43.8 | 35.3 | 60.9 | 26.9
G&D2019 | 15.1 | 11.3 | 23.1 |  07.3
|  PIGLeT | 81.8 | 81.1 | 83.8 | 80.2
(a)

Attribute-level accuracy (Test-Overall,%)
---
size | distance | mass | Temperature | isBroken
8-way | 8-way | 8-way | 3-way | boolean
83.2 | 84.1 | 96.3 | 86.0 | 94.8
73.7 | 77.0 | 89.5 | 84.2 | 94.7
83.9 | 88.9 | 94.3 | 95.4 | 98.1
81.6 | 90.0 | 94.0 | 95.6 | 98.4
81.8 | 84.6 | 94.3 | 96.3 | 95.8
81.1 | 87.5 | 93.6 | 96.1 | 96.5
82.2 | 84.9 | 93.8 | 89.6 | 93.5
87.7 | 87.6 | 97.5 | 93.4 | 97.5
53.4 | 43.6 | 84.0 | 88.1 | 95.1
83.0 | 86.9 | 94.0 | 93.7 | 97.4
68.6 | 47.3 | 82.2 | 88.3 | 95.8
92.3 | 91.9 | 99.2 | 99.8 | 99.0
(b)

Table 1: Overall results. Left: we show the model accuracies at predicting all
attributes of an object correctly. We compare PIGLeT with ‘text-to-text’
approaches that represent the object states as a string, along with BERT-style
approaches with additional machinery to encode inputs or decode outputs.
PIGLeT outperforms a T5 model 100x its size (11B params) and shows gains over
the BERT-style models that also model action dynamics through a language
transformer. Right: we show several attribute-level accuracies, along with the
number of categories per attribute; PIGLeT outperforms baselines by over 4
points for some attributes such as size and distance.

We test our model’s ability to encode language into a grounded form (PIGPeN-
NLU), and decode that grounded form into language (PIGPeN-NLG).

###  4.1 PIGPeN-NLU Results.

We first evaluate models by their performance on PIGPeN-NLU: given objects
$\vec{\bm{o}}$, and a sentence $\bm{s}_{\bm{a}}$ describing an action, a model
must predict the resulting state of objects $\vec{\bm{o}}^{\prime}$. We
primarily evaluate models by accuracy; scoring how many objects for which they
got all attributes correct. We compare with the following strong baselines:

  1. a.

No Change: this baseline copies the initial state of all objects
$\vec{\bm{o}}$ as the final state $\vec{\bm{o}}^{\prime}$.

  2. b.

GPT3-175B Brown et al. (2020), a very large language model for ‘few-shot’
learning using a prompt. For GPT3, and other text-to-text models, we encode
and decode the symbolic object states in a JSON-style dictionary format,
discussed in Appendix A.4.

  3. c.

T5 Raffel et al. (2019). With this model, we use the same ‘text-to-text’
format, however here we train it on the paired data from PIGPeN. We consider
varying sizes of T5, from T5-Small – the closest in size to PIGLeT, up until
T5-11B, roughly 100x the size.

  4. d.

Alberti et al. (2019)-style. This paper originally proposed a model for VCR
Zellers et al. (2019a), where grounded visual information is fed into a BERT
model as tokens; the transformer performs the grounded reasoning. We adapt it
for our task by using our base LM and feeding in object representations from
our pretrained object encoder, also as tokens. Our object decoder predicts the
object, given the LM’s pooled hidden state. This is “pretrained dynamics,” we
also consider a version without a randomly initialized dynamics model.

  5. e.

Gupta and Durrett (2019)-style. Thiso paper proposes using Transformers to
model physical state, for tasks like entity tracking in recipes. Here, the
authors propose decoding a physical state attribute (like isCooked) by feeding
the model a label-specific [CLS] token, and then mapping the result through a
hidden layer. We do this and use a similar object encoder as our Alberti et
al. (2019)-style baseline.

We discuss hyperparameters in Appendix A.3.

Results. From the results (Table 1), we can draw several patterns. Our model,
PIGLeT performs best at getting all attributes correct; doing so over 80% on
both validation and test sets, even for novel objects not seen during
training. The next closest model is T5-11B, which scores 68% on validation.
Though when evaluated on objects ‘seen’ during training it gets 77%, that
number drops by over 18% for unseen objects. On the other hand, PIGLeT has a
modest gap of 3%. This suggests that our approach is particularly effective at
connecting unpaired language and world representations. At the other extreme,
GPT3 does poorly in its ‘few-shot’ setting, suggesting that size is no
replacement for grounded supervision.

PIGLeT also outperforms ‘BERT style’ approaches that control for the same
language model architecture, but perform the physical reasoning inside the
language transformer rather than as a separate model. Performance drops when
the physical decoder must be learned from few paired examples (as in Gupta and
Durrett (2019)); it drops even further when neither model is given access to
our pretrained dynamics model, with both baselines then underperforming ‘No
Change.’ This suggests that our approach of having a physical reasoning model
_outside of_ an LM is a good inductive bias.

####  4.1.1 Ablation study

Model | Accuracy (val;%)
---|---
PIGLeT, No Pretraining | 10.4
PIGLeT, Non-global $\textrm{MLP}_{\textrm{apply}}$ | 72.0
PIGLeT, Global $\textrm{MLP}_{\textrm{apply}}$ | 78.5
PIGLeT, Global $\textrm{MLP}_{\textrm{apply}}$, Gen. loss (6) | 81.8
PIGLeT, Symbols Only (Upper Bound) | 89.3
Table 2: Ablation study on PIGPeN-NLU’s validation set. Our model improves 6%
by modeling global dynamics of all objects in the scene, versus applying
actions to single objects in isolation. We improve another 3% by adding an
auxiliary generation loss.

In Table 2 we present an ablation study of PIGLeT’s components. Of note, by
using a global representation of objects in the world (Equation 4), we get
over 6% improvement over a local representation where objects are manipulated
independently. We get another 3% boost by adding a generation loss, suggesting
that learning to generate summaries helps the model better connect the world
to language. Last, we benchmark how much headroom there is on PIGPeN-NLU by
evaluating model performance on a ‘symbols only’ version of the task, where
the symbolic action $\bm{a}$ is given explicitly to our dynamics model. This
upper bound is roughly 7% higher than PIGLeT, suggesting space for future
work.

###  4.2 PIGPeN-NLG Results

Next, we turn to PIGPeN-NLG: given objects $\vec{\bm{o}}$, and the literal
next action $\bm{a}$, a model must generate a sentence
$\bm{s}_{\vec{\bm{o}}^{\prime}}$ describing what will change in the scene. We
compare with the following baselines:

  1. a.

T5. We use a T5 model that is given a JSON-style dictionary representation of
both $\vec{\bm{o}}$ and $\bm{a}$, it is finetuned to generate summaries
$\bm{s}_{\vec{\bm{o}}^{\prime}}$.

  2. b.

LM Baseline. We feed our LM hidden states $\mathbf{h}_{\bm{o}}$ from our
pretrained encoder, along with its representation of $\bm{a}$. The key
difference between it and PIGLeT is that we do not allow it to simulate
neurally what might happen next – $\textrm{MLP}_{\textrm{apply}}$ is never
used here.

Size matters. Arguably the most important factor controlling the fluency of a
language generator is its size Kaplan et al. (2020). Since our LM could also
be scaled up to arbitrary size, we control for size in our experiments and
only consider models the size of GPT2-base (117M) or smaller; we thus compare
against T5-small as T5-Base has 220M parameters. We discuss optimization and
sampling hyperparameters in Appendix A.3.

Evaluation metrics. We evaluate models over the validation and test sets. We
consider three main evaluation metrics: BLEU Papineni et al. (2002) with two
references, the recently proposed BERTScore Zhang et al. (2020), and conduct a
human evaluation. Humans rate both the fluency of post-action text, as well as
its faithfulness to true action result, on a scale from $-1$ to $1$.

Model | BLEU | BERTScore | Human (test; $[\shortminus 1,1]$)
---|---|---|---
Val | Test | Val | Test | Fluency | Faithfulness
T5 | 46.6 | 43.4 | 82.2 | 81.0 | 0.82 | 0.15
LM Baseline | 44.6 | 39.7 | 81.6 | 78.8 | 0.91 | -0.13
PIGLeT | 49.0 | 43.9 | 83.6 | 81.3 | 0.92 | 0.22
Human | 44.5 | 45.6 | 82.6 | 83.3 | 0.94 | 0.71
Table 3: Text generation results on PIGPeN-NLG, showing models of roughly
equivalent size (up to 117M parameters). Our PIGLeT outperforms the LM
baseline (using the same architecture but omitting the physical reasoning
component) by 4 BLEU points, 2 BERTScore $F_{1}$ points, and 0.35 points in a
human evaluation of language faithfulness to the actual scene.

Results. We show our results in Table 3. Of note, PIGLeT is competitive with
T5 and significantly outperforms the pure LM baseline, which uses a pretrained
encoder for object states, yet has the physical simulation piece
$\textrm{MLP}_{\textrm{apply}}$ removed. This suggests that simulating world
dynamics not only allows the model to predict what might happen next, it leads
to more faithful generation as well.

##  5 Analysis

Figure 4: Qualitative examples. Our model PIGLeT reliably predicts what might
happen next (like the Mug becoming empty in Row 1), in a structured and
explicit way. However, it often struggles at generating sentences for unseen
objects like Mug that are excluded from the training set. T5 struggles to
predict these changes, for example, it seems to suggest that emptying the Mug
causes all containers in the scene to become empty.

###  5.1 Qualitative examples.

We show two qualitative examples in Figure 4, covering both PIGPeN-NLU as well
as PIGPeN-NLG. In the first row, the robot empties a held Mug that is filled
with water. PIGLeT gets the state, and generates a faithful sentence
summarizing that the mug becomes empty. T5 struggles somewhat, emptying the
water from both the Mug and the (irrelevant) Sink. It also generates text
saying that the Sink becomes empty, instead of the Mug.

In the second row, PIGLeT correctly predicts the next object states, but its
generated text is incomplete – it should also write that the mug becomes
filled wtih Coffee. T5 makes the same mistake in generation, and it also
underpredicts the state changes, omitting all changes to the Mug.

We suspect that T5 struggles here in part because Mug is an unseen object. T5
only experiences it through language-only pretraining, but this might not be
enough for a fully grounded representation.

###  5.2 Representing novel words

Figure 5: PIGPeN-NLU performance of a zero-shot PIGLeT, that was pretrained on
Books and Wikipedia without reading any words of our ‘unseen’ objects like
‘mug.’ It outperforms a much bigger T5-11B overall, though is in turn beaten
by PIGLeT on unseen objects like ‘Sink’ and ‘Microwave.’

The language models that perform best today are trained on massive datasets of
text. However, this has unintended consequences Bender et al. (2021) and it is
unlike how children learn language, with children learning novel words from
experience Carey and Bartlett (1978). The large scale of our pretraining
datasets might allow models to learn to perform physical-commonsense like
tasks for wrong reasons, overfitting to surface patterns rather than learning
meaningful grounding.

We investigate the extent of this by training a ‘zero-shot’ version of our
backbone LM on Wikipedia and books – the only difference is that we explicitly
exclude all mentioned sentences containing one of our “unseen” object
categories. In this setting, not only must PIGLeT learn to ground words like
‘mug,’ it must do so without having seen the word ‘mug’ during pretraining.
This is significant because we count over 20k instances of ‘Mug’ words
(including morphology) in our dataset.

We show results in Figure 5. A version of PIGLeT with the zero-shot LM does
surprisingly well – achieving 80% accuracy at predicting the state changes for
“Mug” – despite never having been pretrained on one before. This even
outperforms T5 at the overall task. Nevertheless, PIGLeT outperforms it by
roughly 7% at unseen objects, with notable gains of over 10% on highly dynamic
objects like Toasters and Sinks.

##  6 Related Work

Grounded commonsense reasoning. In this work, we study language grounding and
commonsense reasoning at the representation and concept level. The aim is to
train models that learn to acquire concepts more like humans, rather than
performing well on a downstream task that (for humans) requires commonsense
reasoning. Thus, this work is somewhat different versus other 3D embodied
tasks like QA Gordon et al. (2018); Das et al. (2018), along with past work
for measuring such grounded commonsense reasoning, like SWAG, HellaSWAG, and
VCR Zellers et al. (2018, 2019b, 2019a). The knowledge covered is different,
as it is self-contained within THOR. While VCR, for instance, includes lots of
visual situations about what people are doing, this paper focuses on learning
the physical properties of objects.

Zero-shot generalization. There has been a lot of past work involved with
learning ‘zero-shot’: often learning about the grounded world in language, and
transferring that knowledge to vision. Techniques for this include looking at
word embeddings Frome et al. (2013) and dictionary definitions Zellers and
Choi (2017). In this work, we propose the inverse. This approach was used to
learn better word embeddings Gupta et al. (2019) or semantic tuples Yatskar et
al. (2016), but we consider learning a component to be plugged into a deep
Transformer language model.

Past work evaluating these types of zero-shot generalization have also looked
into how well models can compose concepts in language together Lake and Baroni
(2018); Ruis et al. (2020). Our work considers elements of compositionality
through grounded transfer. For example, in PIGPeN-NLG, models must generate
sentences about the equivalent of dropping a ‘dax’, despite never having seen
one before. However, our work is also contextual, in that the outcome of
‘dropping a dax’ might depend on external attributes (like how high we’re
dropping it from).

Structured Models for Attributes and Objects. The idea of modeling actions as
functions that transform objects has been explored in the computer vision
space Wang et al. (2016). Past work has also built formal structured models
for connecting vision and language Matuszek et al. (2012); Krishnamurthy and
Kollar (2013), we take a neural approach and connect today’s best models of
language _form_ to similarly neural models of a simulated environment.

##  7 Conclusion

In this paper, we presented an approach PIGLeT for jointly modeling language
form and meaning. We presented a testbed PIGPeN for evaluating our model,
which performs well at grounding language to the (simulated) world.

## Acknowledgments

We thank the reviewers for their helpful feedback, and the Mechanical Turk
workers for doing a great job in annotating our data. Thanks also to Zak Stone
and the Google Cloud TPU team for help with the computing infrastructure. This
work was supported by the DARPA CwC program through ARO (W911NF-15-1-0543),
the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen
Institute for AI.

## References

  * Alberti et al. (2019) Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. 2019.  Fusion of detected objects in text for visual question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2131–2140.
  * Artzi et al. (2013) Yoav Artzi, Nicholas FitzGerald, and Luke S Zettlemoyer. 2013.  Semantic parsing with combinatory categorial grammars.  _ACL (Tutorial Abstracts)_ , 3.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big.  _Proceedings of FAccT_.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020.  Experience grounds language.  _arXiv preprint arXiv:2004.10151_.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Carey and Bartlett (1978) S. Carey and E. Bartlett. 1978.  Acquiring a single new word.
  * Das et al. (2018) Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018.  Embodied question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–10.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  _arXiv preprint arXiv:1905.03197_.
  * Frome et al. (2013) Andrea Frome, Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013.  Devise: A deep visual-semantic embedding model.
  * Gordon et al. (2018) Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018.  Iqa: Visual question answering in interactive environments.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Gupta and Durrett (2019) Aditya Gupta and Greg Durrett. 2019.  Effective use of transformer networks for entity tracking.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 759–769.
  * Gupta et al. (2019) Tanmay Gupta, Alexander Schwing, and Derek Hoiem. 2019.  Vico: Word embeddings from visual co-occurrences.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 7425–7434.
  * Harnad (1990) Stevan Harnad. 1990.  The symbol grounding problem.  _Physica D: Nonlinear Phenomena_ , 42(1-3):335–346.
  * Held and Hein (1963) Richard Held and Alan Hein. 1963.  Movement-produced stimulation in the development of visually guided behavior.  _Journal of comparative and physiological psychology_ , 56(5):872.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_.
  * Krishnamurthy and Kollar (2013) Jayant Krishnamurthy and Thomas Kollar. 2013.  Jointly learning to parse and perceive: Connecting natural language to the physical world.  _Transactions of the Association for Computational Linguistics_ , 1:193–206.
  * Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International Conference on Machine Learning_ , pages 2873–2882. PMLR.
  * Matuszek et al. (2012) Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012.  A joint model of language and perception for grounded attribute learning.  In _Proceedings of the 29th International Coference on International Conference on Machine Learning_ , pages 1435–1442.
  * McClelland et al. (2019) James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. 2019.  Extending machine language models toward human-level language understanding.  _arXiv preprint arXiv:1912.05877_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  Technical report, OpenAI.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv e-prints_.
  * Ruis et al. (2020) Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. 2020\.  A benchmark for systematic generalization in grounded language understanding.  _Advances in Neural Information Processing Systems_ , 33.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pages 4603–4611.
  * Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020.  Alfred: A benchmark for interpreting grounded instructions for everyday tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10740–10749.
  * Smith and Gasser (2005) Linda Smith and Michael Gasser. 2005.  The development of embodied cognition: Six lessons from babies.  _Artificial life_ , 11(1-2):13–29.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010. Curran Associates Inc.
  * Wang et al. (2016) Xiaolong Wang, Ali Farhadi, and Abhinav Gupta. 2016.  Actions ~ transformations.  In _CVPR_.
  * Yatskar et al. (2016) Mark Yatskar, Vicente Ordonez, and Ali Farhadi. 2016.  Stating the obvious: Extracting visual common sense knowledge.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 193–198.
  * Yu and Smith (2012) Chen Yu and Linda B Smith. 2012.  Embodied attention and word learning by toddlers.  _Cognition_ , 125(2):244–262.
  * Zellers et al. (2019a) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a.  From recognition to cognition: Visual commonsense reasoning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 93–104, Brussels, Belgium. Association for Computational Linguistics.
  * Zellers and Choi (2017) Rowan Zellers and Yejin Choi. 2017.  Zero-shot activity recognition with verb attribute induction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 946–958.
  * Zellers et al. (2019b) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019b.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, V. Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  _ArXiv_ , abs/1904.09675.

##  Appendix A Model implementation details and hyperparameters.

We discuss the architectures and learning hyperparameters of our various
models in the subsections below.

###  A.1 Physical Dynamics Model

We implemented our dynamics model with three Transformer layers for both the
encoder and the decoder, and a hidden dimension of 256 for objects and
actions. The resulting model has 17 million parameters. We pretrained the
model for 20 epochs over 280k state transitions, with a batch size of 1024. We
use an Adam optimizer Kingma and Ba (2014) with a learning rate of 1e-3.

###  A.2 Ordering attributes in decoding.

Recall that we use a left-to-right transformer to decode into an attribute
representation, predicting attributes one-by-one from $c_{1}$ to $c_{n}$. Our
model is agnostic to the actual order, as no matter what the order is, it
still is modeling a decomposition of the joint probability of generating that
object. However, we implemented this by using the name as the first attribute
$c_{1}$ that is predicted, and ordered the rest in a descending way by
vocabulary size so as to predict harder attributes first.

###  A.3 Optimization Hyperparameters chosen

We finetuned PIGLeT for both tasks with an Adam optimizer Kingma and Ba
(2014). We did a small grid search for hyperparameter values, choosing the
best learning rate $\\{2e^{-5},1e^{-5},1e^{-6}\\}$ by accuracy on the
development set, and likewise the best batch size $16$ or $32$. We considered
freezing the physical dynamics backbone as another hyperparameter. We found it
slightly boosted performance on PIGPeN-NLG when we froze the physical dynamics
backbone, but not so for PIGPeN-NLU. We trained our model for 80 epochs on
paired data.

We trained the baseline models with the same backbone in the same way, using
similar hyperparameters. However, we found that after 80 epochs, the baseline
models without pretrained dynamics failed to converge, so we finetuned them
for 200 epochs total. For T5, we used similar identical hyperparameter ranges
as the other models. However, because T5 uses a different optimizer
(AdaFactor; Shazeer and Stern (2018)), which operates on a slightly different
scale, we used a different set of learning rates. We chose the best one over
$\\{1e^{-4},2e^{-4},4e^{-4}\\}$.

Search. Both of our tasks involve left-to-right decoding. We used argmax
(greedy) search for PIGPeN-NLU, finding that it worked well as a ‘closed-ended
generation’ style task. On the other hand, we used Nucleus Sampling for
PIGPeN-NLG as there are often several ways to communicate a state transition;
here we set $p=0.8$.

###  A.4 Encoding the input for text-to-text models

Text-to-text models, needless to say, can only handle text. We encode the
world states into a representation suitable for these models by formatting the
object states as a JSON-style dictionary of keys and values. We had to make
several modifications to the encoding however from a default JSON, because we
handle a lot of attributes in this task, and JSON has quote characters ‘’‘
that take up a lot of space in a BPE encoding. We thus strip the quote
characters and lowercase everything (with this also helping BPE-efficiency).
We put parentheses around each object and give names to all ‘binned’
attributes.

An example encoding might be:

Predict next object states: (objectname: bowl, parentreceptacles: cabinet,
containedobjects: none, distance: 6 to 8 ft, mass: .5 to 1lb, size: medium,
temp: roomtemp, breakable: true, cookable: false, dirtyable: true, broken:
false, cooked: false, dirty: false, filledwithliquid: false, open: false,
pickedup: false, sliced: false, toggled: false, usedup: false, moveable:
false, openable: false, pickupable: true, receptacle: true, sliceable: false,
toggleable: false, materials: glass) (objectname: egg, parentreceptacles:
none, containedobjects: none, distance: 2 to 3ft, mass: .1 to .2lb, size:
tiny, temp: cold, breakable: true, cookable: true, dirtyable: false, broken:
false, cooked: false, dirty: false, filledwithliquid: false, open: false,
pickedup: true, sliced: false, toggled: false, usedup: false, moveable: false,
openable: false, pickupable: true, receptacle: false, sliceable: true,
toggleable: false, materials: food) (action: throwobject10)

We have models decode directly into this kind of format when predicting state
changes. Though the T5 models usually get the format right, we often have to
sanitize the text in order for it to be a valid object state in our framework.
This is especially an issue with GPT3, since it is given limited supervision
(we squeeze 3 examples into the 2048-BPE token context window) and often makes
up new names and attributes. Thus, for each word not in an attribute’s
vocabulary, we use a Levenstein distance heuristic to match the an invalid
choice with its closest (valid) option. If the model fails to generate
anything for a certain attribute key – for example if it does not include
something like openable somewhere, we copy the representation of the input
object for that attribute, thereby making the default assumption that
attributes do not change.

##  Appendix B All THOR attributes

Attribute Name | Vocab size |  Values
---|---|---
objectName | 126 |  One per object type, along with None
parentReceptacles | 126 |  One per object type, along with None
receptacleObjectIds | 126 |  One per object type, along with None
mass | 8 |  8 bins
size | 8 |  8 bins
distance | 8 |  8 bins
ObjectTemperature | 3 |  Hot, Cold, RoomTemp
breakable | 2 |
canBeUsedUp | 2 |
canFillWithLiquid | 2 |
cookable | 2 |
dirtyable | 2 |
isBroken | 2 |
isCooked | 2 |
isDirty | 2 |
isFilledWithLiquid | 2 |
isOpen | 2 |
isPickedUp | 2 |
isSliced | 2 |
isToggled | 2 |
isUsedUp | 2 |
moveable | 2 |
openable | 2 |
pickupable | 2 |
receptacle | 2 |
sliceable | 1 |
toggleable | 2 |
salientMaterials_Ceramic | 2 |
salientMaterials_Fabric | 2 |
salientMaterials_Food | 2 |
salientMaterials_Glass | 2 |
salientMaterials_Leather | 2 |
salientMaterials_Metal | 2 |
salientMaterials_None | 2 |
salientMaterials_Organic | 2 |
salientMaterials_Paper | 2 |
salientMaterials_Plastic | 2 |
salientMaterials_Rubber | 2 |
salientMaterials_Soap | 2 |
salientMaterials_Sponge | 2 |
salientMaterials_Stone | 2 |
salientMaterials_Wax | 2 |
salientMaterials_Wood | 2 |
Table 4: All attributes that we consider for this work in THOR. We list the
attribute’s name, the size of the attribute vocabulary, and the range of
values the attribute can take on. For attributes like ‘mass’, ‘size’, and
‘distance’, we note that the underlying simulator stores them as floats; we
bin them to 8 values for this work. All the values for attributes with a
vocabulary size of 2 are boolean.

We list a table with all of the attributes we used for this work in Table 4.

##  Appendix C Turk Annotation Details

We followed crowdsourcing best practices, such as using a qualification exam,
giving feedback to workers, and paying workers well (above $15 per hour). Each
of our HITs required writing three sentences, and we paid Mechanical Turk
workers 57 cents per HIT. We used three workers per example, allowing us to
have multiple language references for evaluation. A screenshot of our user
interface is shown in Figure 6.

Figure 6: Our user interface for Mechanical Turk annotation.

Generator |  Description
---|---
put X in Y _ |  Samples an object X from the scene, and a receptacle Y. Tries to put it in Y.
throw X at Y _ |  Samples two objects X and Y from the scene. Picks up X, moves to face Y, and throws it forward with variable intensity.
toggle X _ |  Samples an object X, and turns it on or off.
slice X _ |  Samples an object X and a surface Y. Picks up X, places it on Y, and cuts it.
dirty X _ |  Samples an object X, and makes it dirty.
clean X _ |  Samples a dirty object X. Finds a Sink nearby a Faucet, and places X inside. Turns on/off the Faucet, cleaning X.
toast bread _ |  Finds some Bread, slicing it if necessary, places it in a Toaster, then turns it on.
brew coffee _ |  Picks up a Mug, places it under a CoffeeMachine, and turns the machine on.
fry X _ |  Picks up a food X, slices it if necessary, and puts it in a Pot or Pan. Brings it to a StoveBurner and turns the burner on.
microwave X _ |  Picks up an object X and slices it if necessary. Places it in a Microwave, closes it, and then turns it on.
fill X _ |  Picks up an object X places it under a Faucet. Turns on/off the Faucet, then pours out the liquid.

Table 5: Trajectory generation functions that we used to sample ‘interesting’
physical interactions, such as the effects that actions will have on objects,
and which actions will succeed or not.

##  Appendix D Our Pretrained Language Model

We use our own pretrained language model primarily because it allows us to
investigate the impact of data on model performance. We trained a prefix-
masked language model Dong et al. (2019) on Wikipedia and Book data, mimicing
the data used by the original BERT paper Devlin et al. (2019). We trained the
model for 60000 iterations, at a batch size of 8192 sequences each of length
512. This corresponds to 50 epochs over the dataset. We masked inputs in the
bidirectional prefix with SpanBERT masking Joshi et al. (2020). Since BERT-
style ‘masked’ out inputs are easier to predict than tokens generated left-to-
right, we reduced the loss component of left-to-right generation by a factor
of 20; roughly balancing the two loss components.

Figure 7: Counts of zero-shot words that appear in BERT’s training data
(Wikipedia and Toronto Books). For example, in the 4 billion words BERT is
trained on, it sees the word ‘Bed’ almost 500k times. This might allow it to
perform superficially well at answering questions about beds – while not
necessarily possessing deep physical knowledge about them.