# Deep Convolutional Neural Network for 6-DOF Image Localization

Daoyuan Jia Yongchi Su Chunping Li

Tsinghua University, Beijing, China
jiady14, suyc13, cli @mails.tsinghua.edu.cn

###### Abstract

Wee present an accurate and robust method for six degree of freedom image
localization. There are two key-points of our method, 1). automatic immense
photo synthesis and labeling from point cloud model and, 2). pose estimation
with deep convolutional neural networks (ConvNets) regression. Our model can
directly regresses 6-DOF camera poses from images, accurately describing where
and how it was captured. We achieved an accuracy within 1 meters and 1 degree
on our out-door dataset, which covers about $20,000m^{2}$ on our school
campus. Unlike previous point cloud registration solutions, our model supports
low resolution images (i.e. $224\times 224$ in our settings), and is tiny in
size when finished training. Moreover, in pose estimation, our model uses
$O(1)$ time & space complexity as trainset grows. We will show the importance
to localization using hundreds of thousands of generated and self-labeled
”photos” came from a short video. We will show our model’s robustness despite
of illumination and seasonal variances, which usually fails methods that
leverage image feature descriptors like SIFT. Furthermore, we will show the
ability of transfer our model trained on one scene to another, and the gains
in accuracy and efficiency.

##  1 Introduction

Image localization, or camera relocalization, describes a problem of
approximating the position and orientation of the camera when the query image
was taken. This problem inspires and connects to many cutting-edge
applications in our life, such as self-driving automobiles, unmanned aerial
vehicles (UAVs), virtual reality (VR) and etc.

For years, researchers came to many solutions based on visual descriptors of
images, such as SIFT Lowe (2004) or SURF Bay et al. (2006). We categorize
these methods into three main genres, Descriptors Matching, Dynamic
Reconstruction, and Point Cloud, as well as combinations and optimizations of
them. These solutions usually demand so much computation / storage resource
that invalidates their applications on mobile devices, or restricted to small
scene usage, or require Internet and server-side participation, which is
impractical in some daily scenarios. Our method distinguish from them as we
provide a complete end-to-end solution with deep convolutional neural network
(ConvNet), which directly takes query images as input and regress camera
position and orientation from them.

One of the main problem in exploiting a deep ConvNet is the sparsity of train
samples relative to network’s depth and vast number of parameters. Several
thousands of photos are sufficient to reconstruct a 3D model, but far from for
training a ConvNet to regress camera pose, and would almost certainly lead to
over-fitting. We coped this problem by integrating Point Cloud solution. We
reconstructed the 3D point cloud model of the scene, and from which we can
generate hundreds of thousands of ”photos” [增加了信息], all of which contains
accurate 6-DOF coordinates relative the model. This also solved another big
problem in image localization evaluations we met, which We call it Groundtruth
Dilemma. It is because that the 6-DOF ground truth of test-set and train-set
is not easily available. Datasets like Quad Crandall et al. (2013) and Aachen
Middelberg et al. (2014) were collected with differential GPS to obtain
positions with a precision of $10cm$, and only a handful of test images come
with accurate orientation angles. We captured videos in our school campus and
split them into frames. In total we created 3 [**] scenes, and reconstructed
3D points cloud model for each of them. From each model we generated more than
100,000 accurately labeled ”photos”, of various positions and locations, to
feed our deep ConvNets. We also took advantage of some shaders to augment our
dataset, so that the trained deep ConvNets can tolerate weather and light
variances.

We modified and experimented on CaffeNet Krizhevsky et al. (2012), GoogLeNet
Szegedy et al. (2015) and VGG Simonyan and Zisserman (2014). Due to the depth
of some ConvNets, we modified layer structures and adopted different train
policy. We also found it is very helpful to use pre-trained models, which has
already been trained on other giant dataset like ImageNet Russakovsky et al.
(2015). Pre-trained model serves as a better initialization, accelerated our
training process and produced better converged result. We finally reached an
average accuracy of 1 meters in position and 0.8 degree in orientation. We
extended our experiments to photos in different illumination, season
conditions, and found our model is robust to these changes. We also transfered
our trained model of one scene to another, and found that training on new
scene takes less epochs to converge.

Our contributions are in the following areas.

  * •

we are the first to use synthesis method to generate immense number of
accurately 6-DOF labeled photos from point cloud and shaders, to build and
further extend dataset for camera relocalization. This improves the pose
estimation accuracy of deep ConvNets and robustness to various light, weather
condition;

  * •

we modified and applied three state of art deep ConvNet architectures from
image classification to pose regression and evaluated their result.

The organization of the article is as follow: related work will be covered in
section 2; our pipeline from input scene video to network training is
described in section 3; section 4 provides experimental evaluation and
analysis; we summarize our work in section 6.

##  2 Related work

We categorize current visual descriptors based image localization methods into
three main genres, Descriptors Matching, methods used by Li et al. (2010) and
Li et al. (2009) aimed at estimating a camera pose by matching descriptors to
a known scene. This genre of methods solve localization as a process of image
retrieval by find images in the databases that shared most visual descriptors.
If images in the database are labeled with GPS tags, the location of the query
one can be answered by averaging existed ones. For instance, Schindler et al.
(2007) and Chen et al. (2011) seek to localize urban images by matching images
in Google street-view. Accuracy of methods in this genre is usually coarse to
tens or hundreds of meters in outdoor scene, and seldom supports 6-DOF
localization. Also, as the dataset expands, the query time increase
accordingly. Dynamic Reconstruction, Klein and Murray (2007) and Castle et al.
(2008) utilized simultaneous localization and mapping (SLAM) to concurrently
reconstruct a 3D model and estimate the camera pose from extracted visual
descriptors. As the 3D maps grows larger, the mapping part of the SLAM becomes
more expensive to run on mobile devices. Point Cloud methods Sattler et al.
(2011) and Donoser and Schmalstieg (2014), pre-build a model, using methods
like Structure-from-Motion (SfM) Agarwal et al. (2009) to reconstruct a scene
composed with tens of millions of 3D points. This process requires intensive
computation to extract millions SIFT features from thousands of photos and
find concurrences among them. Moreover, in the query time, it consumes loads
of memory and computing resource to establish 2D-3D correspondences between
scene points and visual features points from query images. Li et al. (2012)
and Sattler et al. (2012) introduced active 3D-2D correspondences match to
improve registration rate and refine the result. Middelberg et al. (2014)
combined SLAM and point cloud solutions together. They incrementally apply
local relative movements measured by SLAM algorithm onto global position
previously calculated by point cloud on the server to obtain current global
position. The decoupling of computation of relative movement and global
position reduces computation on local devices (i.e. mobile devices), but the
computation on the server is still heavy and this pipeline requires network
participation at the very beginning. All these three genres of methods usually
require resources exceeding computation and memory capacities of modern mobile
devices, and thus invalidated their applications.

Apart from this, some researchers focused on regression methods on RGB-D
images. Shotton et al. (2013) used a regression forest to infer
correspondences between each depth image pixels and points in the scene, which
was constructed with RGB-D input images. Their method amended the problem of
too few 2D-3D matches (in-liners correspondences) in point cloud registration
method, as they allow densely or sparsely on points sampling in depth images.
Krull et al. (2014) introduced a two-way procedure to integrate the random
forest regression the distribution generation to improve the generation
ability to deal with occlusion. Their accuracy of 6-DOF pose estimation
exceeded state-of-art methods on certain public datasets. Methods of this type
are usually conducted in small indoor scene and require depth information in
training. In contrast, our method only use a monocular camera, available on
almost every cell phone, to collect necessary data, and evaluations indicate
our method works well on bigger scene.

Convolutional Neural Network (ConvNet) has been proved to be potent in
recognition of many fields, i.e. GoogLeNet and VGG excelled in Ilsvrc in these
two years and are well applied to other recognition tasks. GoogLeNet Szegedy
et al. (2015) introduced ”Inception”, a unit composed of multiple-sized
convolutional / pooling layers. It reduced parameter numbers and enabled
greater depth. VGG Simonyan and Zisserman (2014) exploited very small
convolution filters. They used filters of size $1\times 1$ through $3\times
3$, and stacked several convolutional layers in conjunction. Stacking conv-
layers of small filter is equivalent to a conv-layer of bigger filter. It
would provide better generation ability, and reduced parameters than single
conv-pooling layer with big filter.

Kendall et al. (2015) follows Shotton et al. (2013), and used a modified
GoogLeNet to regress pose directly. They added another layer of size 2048
before final regression layer, and took advantage of model pre-trained on
ImageNet and Flicker datasets. The transfer learning model accelerated the
training on camera relocalization, and decreased the number of training
samples to use. They reached an accuracy of about $3m$ in position and $5deg$
in orientation. We follows the ”transfer” idea and extended choice of deep
ConvNets to CaffeNet, GoogLeNet and VGG. We applied layer-wise training and
fine-tuned with different pre-trained models to reach the best performance.
Moreover, we detailed our method of photo synthesis and dataset augmentation
in different condition.

##  3 Estimate camera pose with ConvNets

In this section we will present our way of 1). automatic generate loads of
labeled data and, 2). modeling the question and ConvNet configurations of
CaffeNet, GoogLeNet, and VGG.

###  3.1 Automatic data generation and labeling

(a) Law school

(b) School stadium

Figure 1: Planar view showing the video frames positions of dataset and
corresponding dense point cloud model of scene law school and school stadium.

In previous camera relocalization regression solutions, photos were regarded
as individual. We think it is beneficial to correlation information of photos
to train a regressors.Based on this intuition, we designed a way to generate
labeled photos does not originally exist in dataset.

(a) Camera translation grid

(b) Camera rotation sphere

(c) Synthesized school stadium photo samples

(d) Synthesized noon school stadium with sky-box

(e) Synthesized dusk school stadium with sky-box and shader

Figure 2: Illustration of camera position/orientation update policy. Fig.2(a)
shows the track of virtual camera movement in scene school stadium. From green
dot to red dot, camera take photos of different orientations at each cross
dot. Fig.2(b) shows the camera rotation. Camera changes its orientation to
each red dot, and take photos. In fig.2(c)-2(e), samples of synthesized photos
in school stadium scene are shown. Photos were rendered with sky-box and
shader to augment reality.

We take input of video, and split it into frames at a frequency of 3Hz. Then,
frames are used to reconstruct dense point cloud model of the scene using SfM.
In implementation, we used VisualSfM Wu (2013) and PVMS Furukawa and Ponce
(2010), which take advantage of GPU acceleration and multicore bundle
adjustment, to fast build the model. In fig.1 we illustrated the bird view of
our frames and corresponding densely reconstructed model.

Dense point cloud model was imported into Unity to synthesize photos. For
simplicity, in Unity we used the coordinate system generated by VisualSfM.
This will not bring trouble because synthesized photos and origin frames
shared the same coordinate system, and it does not matter where the origin was
set. Therefore, we can create a virtual camera in Unity, recalibrate and
synchronize its instincts with our real cell phone. After that, we constantly
1). take a photo with the virtual camera, 2). take down it pose, 3). update it
to a new position / orientation.

The position/orientation update policy is illustrated in fig.2. The track of
virtual camera translation is a grid. At each cross dot, it takes several
photos with various orientation, and record the corresponding 6-DOF
coordinates. Furthermore, we can utilize different shader/sky-box.
Fig.2(c)-2(e) in turn shows the synthesized photos with pure model,
synthesized noon photos rendered sunny sky-box, and synthesized dusk photos
rendered with evening sky-box and shader. This method enables us to simulate
photos of differed time-of-the-day, weather condition and etc.

The synthesis method is meaningful not only because its ability to automatic
generate tons of labeled photos at a short time, but lies in the fact that it
fully utilized information across original video frames. That is, a
synthesized photo may includes details from separated frames, i.e. frame 1 and
frame 1000, determined by the principle of SfM. Using these massive fabricated
labeled data will prevent pose regression deep ConvNet from overfitting and
help ConvNets learn even nuances of spatial variances.

###  3.2 Modeling the question

We represent image $\boldsymbol{I}$ as a matrix of pixels, and its pose
$\boldsymbol{P}$ as a 6 dimension vector, represents as a combination of
position $p$ and orientation $r$.

$\boldsymbol{P}=[p;r]$ (3.1)

Position $p$ is a 3 dimension vector, and orientation $r$ is composed with
pitch, yaw, and roll. We globally use $meter$ and $degree$ as their respective
unit. Our target is to approximate a function $\mathscr{F}$ using ConvNet that
takes $\boldsymbol{I}$ as input and produce $\boldsymbol{P}$. Therefore, we
want to optimize the an objective loss function like:

$\mathscr{L}(\boldsymbol{I})=||\boldsymbol{\mathscr{F}(I)-P}||_{2}$ (3.2)

Since we use a deep ConvNet, we can use an Euclidean loss function taking form
as:

$\mathscr{L^{*}}(\boldsymbol{I})=||\boldsymbol{w}^{T}\boldsymbol{\hat{P}}-\boldsymbol{w}^{T}\boldsymbol{P}||_{2},$
(3.3)

where $\boldsymbol{\hat{P}}$ denotes the predicted pose vector, and
$\boldsymbol{w}$ is the weight factor to balance the importance of position
and orientation error. By default, we use $\boldsymbol{w=1}$. It means that we
regard position error of 1 meter as the same as orientation error of 1 degree.
However, if we can tolerate greater orientation error, greater values can be
applied to corresponding dimension of $\boldsymbol{w}$.

###  3.3 ConvNet Configurations

For complete of comparison, we conducted our experiments on several state-of-
art deep ConvNets, specifically caffeNet, GoogLeNet and VGG16 as our pose
regression network. If only counting layers with trainable parameters,
caffeNet has 8 layers, and composed with traditional convolution-pooling
structure. GoogLeNet has 22 layers, but is organized slightly different. It
introduced ”inception”, a module that consists of horizontal-stacked, various-
sized convolution layers. In GoogLeNet, There are 6 ”inception” module chained
in a row, together with intermediate loss layers to amplify gradient signal.
VGG16 differs from both of them in that it uses very small receptive field of
size $3x3$ through out the whole net, and it stacks several convolutional
layers in conjunction to approximate a conv-layer of greater receptive field.
This makes decision function more discriminative and reduced trainable
parameters.

All three ConvNets takes image of size $224\times 224$ by default. Therefore,
we rescaled and cropped images in dataset to this resolution. We applied
center crop policy to keep the field-of-view, and disabled data mirror, since
mirroring image would confuse the ConvNet by creating ”photos-pose” pair that
should not exist, i.e. in symmetry some scenes. Apart from this, we found it
is important to minus mean value in data layer for final performance.

We made changes to the network settings in a way similar to Kendall et al.
(2015) but with some differences:

  * •

we change the final feature layer size to 6, and increase it learning rate
amplifier;

  * •

we add another full connection layer of right before final feature layer, and
increase it learning rate amplifier;

  * •

for VGG16, we added middle loss layer to amplify gradient signal;

###  3.4 ConvNet Training

We utilized Caffe Jia et al. (2014) to implement our ConvNet training. All
three ConvNets were trained with stochastic gradient decent solver, with the
same train epochs (300), step length of learning rate decrease (30 epochs) and
batch size (90), on a NVidia GTX 980 graphic card. It is necessary to note
here that 4GB graphic card memory is limited for deep ConvNets like GoogLeNet
and VGG using a big batch. Therefore, we used mini-batch to accumulate
gradient over several batches before back-propagation. Instead of training
ConvNets from scratch, we exploited pre-trained models, which has been trained
to stable on ImageNet dataset. Some varied training strategies were applied on
three ConvNets according their instinct characteristics to achieve their best
performance.

  * •

CaffeNet: Base learning rate $10^{-}5$, reducing 90% every step;

  * •

GoogLeNet: Base learning rate $10^{-}6$, reducing 80% every step;

  * •

VGG16: Base learning rate $10^{-}6$, reducing 10%every step.

##  4 Dataset

For this paper, we release an outdoor campus localization dataset, with 2
scenes. This dataset contains the original video file we collected with
iPhone6 cell phone, video frames and dense point cloud model we generated. We
synthesized immense photos with different settings, and included them in the
dataset, too.

This dataset is slightly unusual as it contains two testsets for each scene,
namely testsetA and testsetB. TestsetA is composed with real photos, the
origin video frames, while testsetB consists of synthesized photos, generated
from dense point cloud. Note that testsetA has previously been separated from
origin video frames before 3D reconstruction, and registered to point cloud
afterwards to obtain their 6-DOF label as ground-truth. This guarantees that
there is no intersection between trainset and testsets. We refer frames that
were used to reconstruct 3D model as ”train frames”. The remaining synthesized
photos are used as trainset. Train frames can be included into trainset as an
augmentation, but is not required, since generated photos outnumber train
frames by 2 magnitudes. We illustrated our dataset detail in table 1.

Scene |  Area ($m^{2}$) | TestsetA | TestsetB |  # Train frames |  # Gen. photo
---|---|---|---|---|---
school stadium |  6,000 | 150 | 19k |  1,200 |  109k
Table 1: Dataset details. Scene | Testset | PoseNet* | CaffePose | VGGPose | GoogLePose | GoogLePose†
---|---|---|---|---|---|---
School stadium | TestsetA |  | 3.03m, $8.15^{\circ}$
---
3.77m, $11.0^{\circ}$
4.32m, $8.60^{\circ}$ |  | 1.72m, $1.67^{\circ}$
---
-
| 1.54m, $0.92^{\circ}$
---
2.25m, $1.59^{\circ}$
1.01m, $0.98^{\circ}$
TestsetB |  | 7.33m, $13.0^{\circ}$
---
8.36m, $14.2^{\circ}$
2.32m, $4.51^{\circ}$ | 1.17m, $0.75^{\circ}$ |  | 0.91m, $0.39^{\circ}$
---
1.01m, $1.44^{\circ}$
0.93m, $0.42^{\circ}$
Table 2: 6-DOF regression result.

##  5 Experiments

We will show in this section experimental evaluation of our method pipeline.
For readability, we refers pipeline used by Kendall et al. (2015) as
”PoseNet*”, whose training data directly came from video frames and
coordinates label from SfM result, and regressed result using their modified
version of GoogLeNet. Therefore, we only use train frames and corresponding 7
dimension labels (xyz & rotation quaternion) of our scenes to train a PoseNet*
model.

In contrast, we refers our modification of deep ConvNets, CaffeNet, VGG16 and
GoogLeNet as CaffePose, VGGPose, and GoogLePose respectively in the following
experimental sections. We train these ConvNets with automatic generated photos
and labels proposed sect. 3.1., along with necessary training methods we
mentioned before. The dagger symbol ($\dagger$) is used to denote training the
ConvNet additionally using train frames. i.e. GoogLePose†.

Table 2 lists the pose estimation accuracy in terms of position and
orientation error. We shows that our method is able to accurately localize
images on evaluation scenes. In scenes like School stadium, our proposed
pipeline produced much more accurate result than PoseNet, improves orientation
estimation accuracy to near 10 times and position estimation to 2 times.

## References

  * Agarwal et al. [2009] Sameer Agarwal, Noah Snavely, Ian Simon, Steven M Seitz, and Richard Szeliski.  Building rome in a day.  In Computer Vision, 2009 IEEE 12th International Conference on, pages 72–79. IEEE, 2009.
  * Bay et al. [2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In Computer vision–ECCV 2006, pages 404–417. Springer, 2006.
  * Castle et al. [2008] Robert Castle, Georg Klein, and David W Murray.  Video-rate localization in multiple maps for wearable augmented reality.  In Wearable Computers, 2008. ISWC 2008. 12th IEEE International Symposium on, pages 15–22. IEEE, 2008.
  * Chen et al. [2011] David M Chen, Georges Baatz, Kevin Köser, Sam S Tsai, Ramakrishna Vedantham, Timo Pylvä, Kimmo Roimela, Xin Chen, Jeff Bach, Marc Pollefeys, et al.  City-scale landmark identification on mobile devices.  In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 737–744. IEEE, 2011.
  * Crandall et al. [2013] David Crandall, Andrew Owens, Noah Snavely, and Daniel Huttenlocher.  SfM with MRFs: Discrete-continuous optimization for large-scale structure from motion.  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 35(12):2841–2853, December 2013.
  * Donoser and Schmalstieg [2014] Michael Donoser and Dieter Schmalstieg.  Discriminative feature-to-point matching in image-based localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 516–523, 2014.
  * Furukawa and Ponce [2010] Yasutaka Furukawa and Jean Ponce.  Accurate, dense, and robust multiview stereopsis.  Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(8):1362–1376, 2010.
  * Jia et al. [2014] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell.  Caffe: Convolutional architecture for fast feature embedding.  arXiv preprint arXiv:1408.5093, 2014.
  * Kendall et al. [2015] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2938–2946, 2015.
  * Klein and Murray [2007] Georg Klein and David Murray.  Parallel tracking and mapping for small ar workspaces.  In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on, pages 225–234. IEEE, 2007.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012.
  * Krull et al. [2014] Alexander Krull, Frank Michel, Eric Brachmann, Stefan Gumhold, Stephan Ihrke, and Carsten Rother.  6-dof model based tracking via object coordinate regression.  In Computer Vision–ACCV 2014, pages 384–399. Springer, 2014.
  * Li et al. [2009] Yunpeng Li, David J Crandall, and Daniel P Huttenlocher.  Landmark classification in large-scale image collections.  In ICCV, pages 1957–1964, 2009.
  * Li et al. [2010] Yunpeng Li, Noah Snavely, and Daniel P Huttenlocher.  Location recognition using prioritized feature matching.  In Computer Vision–ECCV 2010, pages 791–804. Springer, 2010.
  * Li et al. [2012] Yunpeng Li, Noah Snavely, Dan Huttenlocher, and Pascal Fua.  Worldwide pose estimation using 3d point clouds.  In Computer Vision–ECCV 2012, pages 15–29. Springer, 2012.
  * Lowe [2004] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * Middelberg et al. [2014] Sven Middelberg, Torsten Sattler, Ole Untzelmann, and Leif Kobbelt.  Scalable 6-dof localization on mobile devices.  In Computer Vision–ECCV 2014, pages 268–283. Springer, 2014.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * Sattler et al. [2011] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Fast image-based localization using direct 2d-to-3d matching.  In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 667–674. IEEE, 2011.
  * Sattler et al. [2012] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Improving image-based localization by active correspondence search.  Computer Vision–ECCV 2012, pages 752–765, 2012.
  * Schindler et al. [2007] Grant Schindler, Matthew Brown, and Richard Szeliski.  City-scale location recognition.  In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pages 1–7. IEEE, 2007.
  * Shotton et al. [2013] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon.  Scene coordinate regression forests for camera relocalization in rgb-d images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2930–2937, 2013.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2015.
  * Wu [2013] Changchang Wu.  Towards linear-time incremental structure from motion.  In 3D Vision-3DV 2013, 2013 International Conference on, pages 127–134. IEEE, 2013.


# Implicit causality in GPT-2: a case study

Hien Huynh  Tomas O. Lentz  Emiel van Miltenburg
Department of Communication and Cognition
Tilburg University
{m.h.huynh_1, t.o.lentz<EMAIL_ADDRESS>

###### Abstract

This case study investigates the extent to which a language model (GPT-2) is
able to capture native speakers’ intuitions about _implicit causality_ in a
sentence completion task. We first reproduce earlier results (showing lower
surprisal values for pronouns that are congruent with either the subject or
object, depending on which one corresponds to the implicit causality bias of
the verb), and then examine the effects of gender and verb frequency on model
performance. Our second study examines the reasoning ability of GPT-2: is the
model able to produce more sensible motivations for why the subject verbed the
object if the verbs have stronger causality biases? We also developed a
methodology to avoid human raters being biased by obscenities and disfluencies
generated by the model.

aboveexskip=1ex,belowexskip=1ex

Implicit causality in GPT-2: a case study

Hien Huynh and Tomas O. Lentz and Emiel van Miltenburg Department of
Communication and Cognition Tilburg University {m.h.huynh_1, t.o.lentz,
<EMAIL_ADDRESS>

##  1 Introduction

This paper is a case study, highlighting different ways to analyse the
linguistic abilities of a language model, with respect to an established
linguistic phenomenon, namely Implicit causality (IC) bias Hartshorne (2014).
Speakers associate either the subject or the object of a verb with the cause
of the state or event described by that verb. For example, the verb frighten
is a _subject-biased_ verb because native speakers of English tend to see the
subject as the cause of the frightening event. Thus, given a main clause like
in (1), participants in a sentence completion task would tend to provide a
reason referring to the subject, as in (1).

labeloffset=5px,textoffset=5px [main clause John scared Mary because …][reason
he put on a Halloween costume.]

Earlier work by Upadhye et al. (2020) and Davis and van Schijndel (2020, 2021)
investigated the extent to which language models are able to capture native
speakers’ IC biases. This paper aims to reproduce some of their earlier
results, using GPT-2 Radford et al. (2019) as an example. Using the same
sentence completion task as Davis and van Schijndel (2020), we further
investigate how bias, subject gender, and verb frequency influence the
behavior of this model (§3). Next, we will more thoroughly assess the quality
of the completions generated by GPT-2 (§4), asking: does the model’s
performance hold up to further scrutiny?

#### Why GPT-2?

Although it is neither the most recent, nor the best performing open-source
language model around (see Black et al. 2022 for alternatives), GPT-2 is still
a very popular choice for many researchers and practitioners.111See
https://huggingface.co/gpt2 for statistics. This popularity is at least partly
due to its size, as the model can be run and fine-tuned on consumer
hardware.222The model’s popularity means that studying its capabilities and
limitations may be more impactful (at least in the short term) than studying
the capabilities and limitations of larger but less accessible systems. For
us, GPT-2 offers the right balance of complexity and efficiency; as shown by
Upadhye et al. (2020), its outputs are good enough to have a meaningful
discussion about the assessment of language model performance, without
requiring a large (and expensive) computational infrastructure.

#### Contributions

Next to the value of reproducing earlier work, and providing further details
on the IC-related behaviour of GPT-2, the main innovation of this paper is the
controlled assessment of output quality. We took great care to separate issues
with the fluency and offensiveness of the output from the content of the
generated continuations.333See Appendix A for the limitations of this study.
This not only makes the task less harmful to our participants, but it also
increases their focus on our construct of interest: the reasoning abilities of
language models.

##  2 Data

We present two studies. Study 1 investigates next-word surprisal values, and
Study 2 looks at continuations that are generated based on a prompt. For both
our studies, we provide the model with input sentences of the following form:

Subject verb-ed object because ….

The verbs are derived from a list of 246 IC verbs compiled by Ferstl et al.
(2011), who also provide a _bias score_ for each of these verbs. This score is
derived from a human experiment, where participants were asked to complete
sentences like (1). The human bias scores range from -100 (i.e., all valid
continuations produced by respondents in Ferstl et al.’s experiment uniquely
referred to objects of the preceding clauses) to 100 (i.e., all valid
continuations referred to subjects of the preceding clauses).

The subjects and objects are provided by Davis and van Schijndel (2020), who
produced a list of 14 noun pairs that are grammatically male and female (e.g.
_man, woman_ or _brother, sister_). A combination of the nouns and verbs, our
set of stimuli consists of 6888 examples (246 verbs × 14 pairs of gender-
mismatched nouns × 2 subject genders).

##  3 Study 1: IC-bias and pronoun use

First, we investigate whether verbs in GPT-2 possess the same subject/object
bias as in the human experiment described above.

#### Set-up

To test the hypothesis, we use the approach from Davis and van Schijndel
(2021) and checked for each prompt whether the model assigned a lower
surprisal to a male or female pronoun (i.e.  _he_ or _she_).444Next-word
surprisal is estimated for a target upcoming word by taking the inverse log of
this word’s probability:
$\textit{surprisal}(w_{t})=-\textit{log}P(w_{t}|w_{1}\ldots w_{t-1})$ If GPT-2
captures the IC bias, then subject-biased verbs with a female subject should
lead the model to produce lower surprisal values for _she_. Since each noun
pair is used in both orders (either a male or a female noun in subject
position), we have a perfectly balanced dataset.

#### Results

Figure 1 shows the results split by subject gender and bias scores from Ferstl
et al. (2011). GPT-2 generally picks up on the subject or object bias of the
verb. The gender produces more subject-based explanations if the verb’s IC
bias is more subject-biased. There is only one exception: Performance for
sentences with both subject-biased verbs and female subjects is at chance
level for all bias scores.

Figure 1: Heatmap table showing the percentage of outputs matching the subject
or object bias of the verb, with scores separated by subject gender.

IC Bias and Gender. We used the lme4 Bates et al. (2015) and lmerTest package
in R Kuznetsova et al. (2017) to carry out a mixed effects regression analysis
of the relationship between the bias scores and GPT-2’s subject-preference
scores, corresponding to the difference between the surprisal for the object-
congruent pronoun and the surprisal of the subject-congruent pronoun. Details
about our statistical analyses are provided in Appendix B. There were
significant fixed effects of human bias score
($b=0.003,\textrm{SE}=0.0001,p<.001$) and of the interaction between human
bias score and subject gender ($b=0.0014,\textrm{SE}=0.0002,p<.001$). This
confirms our observations from Figure 1. The effect of subject gender was
found to be not significant ($b=-0.0103,\textrm{SE}=0.012,p=.374$).555In
addition, the coefficient of determination of the model was calculated based
on the method developed by Nakagawa and Schielzeth (2013) using the MuMIn
library in R Bartoń (2022). Approximately 14% of the variance in the GPT-2’s
subject-preference score could be explained by the fixed effects alone
(marginal R2 = .142) while 23% of the variance in the subject-preference score
could be explained by both fixed and random effects (conditional
$R^{2}=.230$).

Verb frequency and model performance. We then investigated whether verb
frequency is positively correlated with the language model performance. For
more frequent verbs, we hypothesize that the model more closely matches the
human subject/object bias. To test this hypothesis, we regressed the squared
errors from the previous mixed effects model to the log-transformed word
frequencies of verbs used in the stimuli. As a proxy for verb frequency in the
training corpus, we used the SUBTLEX-US word frequencies from Brysbaert and
New (2009). We found a significant fixed effect of log-transformed word
frequencies of the verbs in our materials
($b=-0.049,\textrm{SE}=0.012,p<.01$). This supports our hypothesis: More data
leads to a better approximation of human IC bias, in terms of the preference
for either male or female pronouns.

##  4 Study 2: Assessing continuations

We now turn to continuations generated by GPT-2 based on the prompts described
in Section 2. Our results above suggest that GPT-2 can generate continuations
according to the causality bias pattern, given enough data. Based on this
result, we now hypothesize that such continuations are better when the IC
pattern is clearly present in the data. The stronger the IC bias is, the
clearer the pattern of continuations in the training set. Of course, this task
is much more difficult than simply generating the right pronouns. We now want
to see whether the continuations actually _make sense_ in the eyes of human
judges. To this end, we collected human ratings for a carefully controlled
subset of the continuations generated by GPT-2.666For a detailed description
of the full set of continuations, see Appendix C. IRB approval was obtained
prior to this study. See Appendix 7 for our ethical considerations.

#### Participants

We used the Prolific777https://www.prolific.co participant pool to recruit 75
participants for the sentence rating task. Our items were spread across 25
different lists, and each participant was only allowed to provide ratings for
one list. We restricted potential participants to native speakers of English,
either from the UK or from the USA. After assessing response quality, we
recruited five additional participants, to obtain three reliable judgments per
item.

#### Task and target construct

Each continuation was assessed by three different participants, who judged
whether the continuations were reasonable, given the prompt. We set up our
experiment as a rating task, where each participant indicated for a list of 40
items whether they agreed with the statement that the continuation was
‘reasonable.’ Participants could indicate their agreement on a five-point
Likert scale, ranging from ‘Strongly Agree’ to ‘Strongly Disagree.’ With the
addition of some examples in our task description (see Appendix H), we
targeted our participants’ intuitions for what makes a good reason to do
something. As we will discuss below, we aimed to avoid any influence of the
form of the output as much as possible.888Given the terminological confusion
in the field of Natural Language Generation Howcroft et al. (2020), Belz et
al. (2020) developed a categorization system for evaluation criteria in NLG.
In terms of their taxonomy, our informal notion of ‘reasonable continuation’
clearly focuses on the content of the output, but the frame of reference is
harder to define. If we look at the completed sentence as a whole, the
sentence is evaluated in its own right, and so it is a question of
_Coherence_.

#### Prompt selection

Due to financial limitations, we were not able to obtain ratings for all 6888
continuations. Following the recommendations from van Miltenburg et al.
(2021), we used a stratified sampling approach. We selected the 5 most
frequent noun pairs, and the 10 most frequent verbs for each of the 10
different bias levels (as illustrated in Figure 1). This selection gives us a
sense of the upper bound performance with respect to continuation quality.
Frequency was again determined using the SUBTLEX-US data Brysbaert and New
(2009). Prompts were constructed in the same way as before (see Ex. 2), with
each noun pair being presented in both orders. This yields 10 verbs × 10 bias
levels × 5 nouns × 2 orders = 1000 prompts.

#### Data preparation

We used GPT-2 to generate continuations for each prompt. As shown earlier, the
problem with these continuations is that they may be offensive or contain
disfluencies (most notably repetition, see Fu et al. 2021). This creates two
problems: (i) offensive output may cause psychological harm for our
participants, (ii) offensiveness and disfluencies may lower the reliability of
the ratings, if participants consistently provide lower scores for
offensive/disfluent outputs (even though they may be consistent with the
prompt). To prevent harm, and to avoid noise in the ratings, we took the
following approach:

  1. 1.

If the output is offensive, select a different noun pair from the 9 remaining
pairs to generate a non-offensive alternative continuation.

  2. 2.

If the output contains repetition, manually remove repeated elements from the
sentence, so that the core content of the continuation remains largely
unchanged.

#### Reliability

To assess annotator reliability, we used a leave-one-out approach to correlate
each participant’s scores with the mean scores of the two other participants
who rated the same responses. Initial correlations ranged between 0.17 and
0.84. Five annotators scored below our cutoff of 0.4, and thus we recruited
five more participants. After re-computing the reliability scores, we kept
only the three highest-scoring participants per task. This way, we obtained a
mean score of 0.64, with a standard deviation of 0.11. This is a strong
correlation, considering the subjective nature of the task.

Rating | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Raw counts | 689 | 571 | 333 | 782 | 625
% of ratings | 23 | 19 | 11 | 26 | 21
% avg rating >= rating | 100 | 79 | 55 | 32 | 7
Table 1: Distribution of the ratings. Ratings correspond to a Likert scale,
where 1=Strongly disagree, 2=Somewhat disagree, 3=Neither agree nor disagree,
4=Somewhat agree, 5=Strongly agree.

#### Continuation quality

Table 1 shows the ratings in three different ways. The top row shows the
(bimodal) raw score distribution: ratings tend to be either negative or
positive, with relatively few ratings in the middle of the scale. The second
row provides the same values as percentages, as a guide for the third row. The
third row shows the percentage of continuations for which the average rating
is greater than or equal to a given rating. For example: only 32% of all the
continuations have an average rating greater than or equal to 4.

A superficial analysis of the lowest and highest rated sentences suggests that
the lowest ratings are often given to sentences in which swapping subject and
object would improve the continuation. For example (see Appendix E for more):

The woman surprised the man because he was wearing a black suit and a black
tie.

Future research could look into carrying out a more systematic error analysis,
to create a taxonomy and present the distribution of the different kinds of
errors in the continuations generated by the model van Miltenburg et al.
(2021).

#### Explaining model performance

We again used a mixed effects model to analyze our results. Our aim is to
explain GPT-2’s performance (i.e., how reasonable the continuation is) in
terms of verb frequency and absolute IC bias (which only looks at strength of
the bias).999IC bias and log-transformed frequency values were determined as
for Study 1. IC bias scores were made absolute; the mean absolute IC bias is
49.3 (SD 27.5, range [2, 92]). The mean (untransformed) frequency per million
is 45.4 (SD 90.7, range [0.02, 502.27]). The correlation between IC bias and
frequency was not significant and low (Pearson’s $r=-0.02,p=.35$). Full
details about the model and model fitting are provided in Appendix D.

Subject gender and its interactions were considered, but did not significantly
improve model fit, and were dropped. Absolute IC bias has a small but positive
effect on the rating ($b=0.003,\textrm{SE}=0.001,p=0.015$). The effect of
frequency is negative ($b=-0.121,\textrm{SE}=0.038,p=0.002$). There is also a
negative interaction of bias and frequency
($b=-0.006,\textrm{SE}=0.001,p<.001$), indicating that the positive effect of
bias diminishes for higher frequency verbs. Thus, while we concluded in Study
1 that the accuracy of GPT-2 with respect to subject/object bias increases as
the verb becomes more frequent, we now find that higher frequency does not
give us more reasonable continuations.

##  5 Discussion

The difference between Study 1 and Study 2 indicates a qualitative difference
between generating pronouns and providing explanations: The latter requires
higher-level reasoning which may not be present in language models like GPT-2
(Bender and Koller, 2020). Though the fact that GPT-2 follows the implicit
causality bias in pronoun selection is at least compatible with knowledge of
causality, the quality of the continuations suggests the pronoun selection is
based on superficial heuristics rather than a deep understanding of language
(also discussed as _fast_ versus _slow_ ; see Choudhury et al. 2022; Kahneman
2011). Although existing suites for LM evaluation (e.g. Ettinger 2020) are
useful, slower forms of assessment (such as human evaluation) are helpful to
tease out this difference.

##  6 Conclusion

This paper showed two different ways to assess the linguistic capacity of a
language model (GPT-2), with respect to one particular cognitively meaningful
pattern, namely implicit causality. The techniques used above can be applied
in a black box setting, without the need to look at the internals of the
model. We hope that this paper is useful for others aiming to assess the
ability of other language models to capture different linguistic phenomena as
well. Our findings also showed that automatic assessment methods may not be
enough to determine whether semantic phenomena like implicit causality are
learned by a language model. Human evaluation remains a necessary complement
to automatic evaluation van der Lee et al. (2021). Our paper shows one way to
do this without participants being influenced by factors like grammaticality
and offensiveness of the output.101010All code and data from this paper is
available at REDACTED FOR REVIEW.

##  7 Ethical considerations

Because our study deals with human subjects, we first obtained ethical
approval from our IRB. We describe our considerations below.

###  7.1 Information letter and informed consent

Our IRB mandates the use of a separate information letter (Appendix F) and
informed consent form (Appendix G). With the information letter, we give a
general description of the study, and provide an indication of potential risks
and benefits of the study. The informed consent form is provided separately to
prevent information overload, and to ensure that participants know what they
are agreeing to, if they decide to take part in our study.

###  7.2 Crowdsourcing and payment

Crowdsourcing has been criticized for its potentially exploitative nature Fort
et al. (2011). We explicitly frame our task as an _experiment_ with _human
participants_ , rather than a _human intelligence task_ with _crowdworkers_ ,
and apply the same considerations and protections as for lab experiments.
Nevertheless, it is still work, and work needs to be paid. Based on
experience, we expected participants to spend roughly 15 minutes on the task,
and set the compensation to £2.40, which amounts to £9.60/hour; 10 cents above
the current UK minimum wage.111111See
https://www.gov.uk/government/publications/the-national-minimum-wage-in-2022
In the end, the vast majority of our participants spent less time than
expected on our task (range:, mean:, standard deviation:). All participants
were compensated for their time, including those providing low-quality
responses.

###  7.3 Offensive material

We wanted to avoid confronting our participants with profanity or otherwise
potentially harmful language. We manually identified potentially offensive
continuations generated by the model, and replaced harmful outputs with
alternative continuations generated for different prompts. We considered
continuations potentially harmful if they contained profanity or made
reference to religion, violence, or sexual acts. All originally generated
sentences and their replacements can be found in the GitHub repository
associated with our paper.

###  7.4 Language model-related harms

Language models are associated with several different harms Bender et al.
(2021); Weidinger et al. (2021), but these harms also depend on the task at
hand. For example, since we used a pretrained model, our study did not incur
any additional training costs. And as described above, since no one other than
the authors were directly exposed to the model’s output, we could prevent our
participants from seeing harmful or toxic content. Thus we are mostly left
with inference costs, which are relatively low, since GPT-2 can run on a
personal computer.

###  7.5 Intended use of this work

This study serves two purposes: (i) to explore the ability of a language model
(GPT-2) to capture native speakers’ intuitions about implicit causality, and
(ii) to develop an evaluation methodology that isolates coherence of the
responses from other factors like offensiveness and (un)wellformedness. We do
not wish to make any claims about the cognitive capacity of language models in
general, nor do we want to claim that GPT-2 can somehow reason about the
world. We just want to see whether the model can generate outputs that follow
earlier observations about implicit causality, and that are internally
consistent. Follow-up studies in the spirit of this work are encouraged.

###  7.6 Licensing

All resources used for this study were developed for research purposes, but
not all materials have a clearly indicated license. GPT-2 is provided under
the MIT license.121212https://huggingface.co/gpt2 The work by Davis and van
Schijndel (2020); Davis et al. (2021) is provided on GitHub without any
license131313https://github.com/forrestdavis/ImplicitCausality and both
Brysbaert and New (2009) and Ferstl et al. (2011) published their work in the
_Behavior Research Methods_ journal without a clear license, but with a clear
intention for their work to be used for research purposes. We thus conclude
that academic use of these resources is warranted.

## References

  * Bartoń (2022) Kamil Bartoń. 2022.  _MuMIn: Multi-Model Inference_.  R package version 1.46.0.
  * Bates et al. (2015) Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015.  Fitting linear mixed-effects models using lme4.  _Journal of Statistical Software_ , 67(1):1–48.
  * Belz et al. (2020) Anya Belz, Simon Mille, and David M. Howcroft. 2020.  Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 183–194, Dublin, Ireland. Association for Computational Linguistics.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 95–136, virtual+Dublin. Association for Computational Linguistics.
  * Brysbaert and New (2009) Marc Brysbaert and Boris New. 2009.  Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english.  _Behavior research methods_ , 41(4):977–990.
  * Choudhury et al. (2022) Sagnik Ray Choudhury, Anna Rogers, and Isabelle Augenstein. 2022.  Machine reading, fast and slow: When do models "understand" language?
  * Davis et al. (2021) Fineen Davis, Eddie Antonio Santos, and Heather Souter. 2021.  On the computational modelling of Michif verbal morphology.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2631–2636, Online. Association for Computational Linguistics.
  * Davis and van Schijndel (2020) Forrest Davis and Marten van Schijndel. 2020.  Discourse structure interacts with reference but not syntax in neural language models.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 396–407, Online. Association for Computational Linguistics.
  * Davis and van Schijndel (2021) Forrest Davis and Marten van Schijndel. 2021.  Uncovering constraint-based behavior in neural models via targeted fine-tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1159–1171, Online. Association for Computational Linguistics.
  * Derczynski et al. (2022) Leon Derczynski, Hannah Rose Kirk, Abeba Birhane, and Bertie Vidgen. 2022\.  Handling and Presenting Harmful Text.  _arXiv e-prints_ , page arXiv:2204.14256.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Ferstl et al. (2011) Evelyn C Ferstl, Alan Garnham, and Christina Manouilidou. 2011.  Implicit causality bias in english: A corpus of 300 verbs.  _Behavior Research Methods_ , 43(1):124–135.
  * Fort et al. (2011) Karën Fort, Gilles Adda, and K. Bretonnel Cohen. 2011.  Last words: Amazon Mechanical Turk: Gold mine or coal mine? _Computational Linguistics_ , 37(2):413–420.
  * Fu et al. (2021) Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021.  A theoretical analysis of the repetition problem in text generation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(14):12848–12856.
  * Hartig (2022) Florian Hartig. 2022.  _DHARMa: Residual Diagnostics for Hierarchical (Multi-Level / Mixed) Regression Models_.  R package version 0.4.5.
  * Hartshorne (2014) Joshua K. Hartshorne. 2014.  What is implicit causality? _Language, Cognition and Neuroscience_ , 29(7):804–824.
  * Hashimoto et al. (2019) Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019.  Unifying human and statistical evaluation for natural language generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1689–1701, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Kahneman (2011) Daniel Kahneman. 2011.  _Thinking, fast and slow_.  Farrar, Straus and Giroux, New York.
  * Kuznetsova et al. (2017) Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017.  lmertest package: Tests in linear mixed effects models.  _Journal of Statistical Software_ , 82(13):1–26.
  * Nakagawa and Schielzeth (2013) Shinichi Nakagawa and Holger Schielzeth. 2013.  A general and simple method for obtaining r2 from generalized linear mixed-effects models.  _Methods in ecology and evolution_ , 4(2):133–142.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  Technical report, OpenAI.
  * Upadhye et al. (2020) Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020.  Predicting reference: What do language models learn about discourse models? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 977–982, Online. Association for Computational Linguistics.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021\.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * van Miltenburg et al. (2021) Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, and Luou Wen. 2021.  Underreporting of errors in NLG output, and what to do about it.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 140–153, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * van Miltenburg et al. (2018) Emiel van Miltenburg, Desmond Elliott, and Piek Vossen. 2018.  Measuring the diversity of automatic image descriptions.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1730–1741, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021.  Ethical and social risks of harm from language models.  _CoRR_ , abs/2112.04359.
  * Zhang et al. (2021) Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021.  Trading off diversity and quality in natural language generation.  In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_ , pages 25–33, Online. Association for Computational Linguistics.

##  Appendix A Limitations

The main limitation of our paper is that we focused on only one language model
(GPT-2), and only in one language (English). So while our findings provide
insights into the capacities of the English GPT-2, they cannot be generalised
to other language models or other languages (which is also illustrated by
Davis and van Schijndel (2021)). Our main contribution is methodological,
namely exploring how to assess the linguistic capacities of language models.
Because our approach treats GPT-2 (mostly) as a black box, our analysis can
easily be applied to other models as well. For other models, different results
may be obtained.

For Study 2, a negative effect of verb frequency on the quality of generated
continuations was found. As the materials used for the rating study were not
gathered to cover the full range of frequencies, this pattern should not be
generalized and may reflect a hidden effect. The negative interaction with IC
bias suggests the same. The model fit shows an imperfect fit, but without one
clear deviation from linearity. Hence, our findings may well be explained
better with more appropriate independent variables.

##  Appendix B Study 1: Statistical analysis

As noted in Section 3, we used the lme4 Bates et al. (2015) and lmerTest
packages in R Kuznetsova et al. (2017) to carry out a mixed effects regression
analysis of the relationship between the model’s subject-preference score and
human bias score of every IC verb, reducing the complexity by removing terms
that do not significantly improve fit.

As fixed effects, human bias score, subject gender of the stimulus sentence-
fragments (with two levels, namely male or female) and their interaction were
included. Moreover, we included item, which indicated the pairs of antonymous
nouns used in the stimulus sentences as subjects and objects, as a random
effect, and added by-item random slopes for the effects of human bias score,
subject gender and their interaction. However, the fitting of the full model
including all fixed and random effects failed to converge. Therefore, the by-
item random slopes for the effects of human bias, subject gender and their
interaction were removed. Our final model is the following in R notation:

subject-preference score ~ human IC bias score * subject gender + (1|item)

We regressed the squared errors obtained from the linear mixed effects
regression model we performed in the previous step on the log-transformed word
frequencies of IC verbs used in the stimuli. As random effects, item was also
entered into the model, and by-item random slope for the effect of log-
transformed frequency was added.

##  Appendix C Study 2: Continuations

While Study 1 looked into the surprisal values for the generation of
male/female pronouns, this study looks at continuations themselves.

###  C.1 Consistency check

We first checked whether the IC bias pattern continues to be present in the
full continuations. In all but 176 cases, the model generated a gendered
pronoun. These cases were coded manually, to determine whether the
continuation referred to the subject or the object. Figure 2 shows the
proportion of continuations referring to either the subject or the object of
the prompt (Y-axis), split by the bias score (X-axis). We see the same trends
as in the first study: overall we see that references to the subject increase
as the bias score increases. This trend also holds when we split the prompts
by subject gender, but for the female subjects the proportion of references to
the subject never exceeds 50%.

Figure 2: Percentage of continuations referring to subjects or objects by bias
scores of verbs. Panel A shows overall proportion, Panel B shows proportion
split by subject gender

###  C.2 Patterns

We then inspected the frequencies of the continuations. We observe that more
than 20% of the outputs is repeated more than 100 times. This lack of
diversity is a common issue in (neural) Natural Language Generation (e.g. van
Miltenburg et al. 2018; Hashimoto et al. 2019; Zhang et al. 2021). It is not
necessarily a problem at the individual level (the continuation may be a bit
generic but still appropriate for the given context), but at the corpus level
these ‘one size fits all’ continuations are shortcuts that prevent more varied
outputs.141414We might also question whether it is possible at all to properly
assess the cognitive capacity of a model that keeps using such shortcuts
(which may be seen as cheating). As an alternative, we can imagine a two-stage
process where researchers first generate an unrestricted set of continuations,
and then force the model to avoid common continuations.

###  C.3 Offensive output

Looking over the generated outputs, there were several occasions where the
model generated offensive continuations, including instances of sexism,
racism, and misogyny. On top of this, the model outputs also contained
sexually explicit words, and some continuations described acts of violence.
Following the recommendations from Derczynski et al. (2022), we do not provide
any examples in this paper. As discussed in Section 4, we also removed
(potentially) offensive continuations from our human rating experiment. For
transparency reasons, we do provide those sentences in our GitHub repository.

Male subject |  | Female subject
---|---|---
Continuations | Frequency |  | Continuations | Frequency
he was afraid of her | 521 |  | she was afraid of him. | 476
she was a woman | 102 |  | he was a good man and he was a good man | 166
he was a good man and he was a good man | 68 |  | he was a good man | 124
she was a good girl | 64 |  | he was a good boy | 92
she was a woman, and he was a man | 60 |  | she was a woman | 57
Table 2: Most frequent continuations for the prompts in Study 2, split by
subject.

###  C.4 Repetitions

Table 2 shows the five most frequent continuations for our prompts, split by
subject gender. It is clear from the table that the continuations generated by
GPT-2 are very repetitive, and tend to be generic without any specific
details.

##  Appendix D Study 2: Statistical analysis

A full linear model was built with the factors subject gender, absolute IC
bias (centered), log-transformed word frequency (z-scored) and their
interactions as fixed effects, and random effects for participant (intercept,
and slopes for the three main effects), sentence (intercept), and
subject/object pair (intercept). The model was then subjected to the step
function of lmerTest, which removes insignificant components. Though the full
model and some of the first reductions of it did not converge, this procedure
is still appropriate (a model that does not converge should not be used, and
as more data is not available, the model should be simplified). The reduced
final model, that did converge, is, in R syntax:

rating ~ abs(IC bias_abs_c) * frequency + (1 | sentence_ID) + (1|participant) + (1|item)

The DHARMa package Hartig (2022) was used to assess model fit. Though the
residuals are not normally distributed, the deviations show no clear pattern.
To avoid spurious conclusions, we corroborated the significant estimates by
checking if their 95% confidence intervals included 0, which they did not.
Hence, treating ratings as a continuous numeric variable was not problematic.

Figure 3: Ratings by absolute value of human IC bias. Figure 4: Ratings by
absolute value of human IC bias, separated for four different levels of verb
frequency.

##  Appendix E Study 2: Details about the ratings

A total of 72 sentences was only rated at the lowest level. Three random
examples are:

  1. 1.

The man fed the woman because he was afraid of her.

  2. 2.

The woman worried the man because he was a bigger man.

  3. 3.

The woman surprised the man because he was wearing a black suit and a black
tie.

Tentatively, we interpret these sentences as a mix-up of two patterns. The
first sentence continues with the subject, but gives no good explanation (in
fact, being afraid of someone would not make feeding them a good idea). The
second and third example continue with an object explanation, but would make
more sense if they had continued with a subject explanation, i.e., flipping
the gender, e.g., ‘The woman surprised the man because _she_ was wearing …’.

There were 69 continuations that always received the highest rating
tentatively seem to conform to the human IC bias. One random example is ‘The
woman thanked the man because he was a good man.‘; the verb ‘to thank’ is
strongly object-biased (raw subject bias score -92).

##  Appendix F Information letter

Title: Assessing computer-generated texts

Introduction
We invite you to take part in our study to assess computer-generated texts.
This study is part of a larger project to see how good or bad computers are at
producing or understanding human language,such as English. In this study you
will be asked to rate the quality of computer-generated texts/sentences. We
will use this information to see what the computer is good at, and to see
where it can still be improved.

What do I have to do?
As mentioned above, you will be asked to rate the quality of computer-
generated texts. In this study, you will be asked to read 40 short sentences,
and to provide your judgment. We are interested in your intuition as a native
speaker of English, so you don’t need to think too long about it.

Expected duration
We expect this study to take about fifteen minutes of your time. Other than
this, we do not foresee any risks associated with this study. On the positive
side, your participation will improve our understanding of the language
capacity of modern computer models.

Ethics and rights
This study was approved by the “Research Ethics and Data Management Committee”
of REDACTED.

Your participation is completely voluntary. Your consent to participate
generally applies for the duration of this study. However, you have the right
to decline to participate and withdraw from the research once participation
has begun, without any negative consequences, and without providing any
explanation.

Your participation is completely anonymous. We will not store any identifying
information, so your answers cannot be traced back to you. We only see the
demographic information that Prolific provides. Do let us know if you would
like to have a copy of your responses, and we will try to obtain them based on
the Prolific ID.

Use of data
Your responses will be used for the current study, and possible follow-up
studies in the future. This means that the data will be presented in research
articles, that are publicly available. For full transparency, we will also
publicly share the anonymised individual responses. As such, they will be
stored indefinitely.

Contact
If you have any questions, or if would like to learn more about this study,
please contact REDACTED for any of your questions.

Ethics approval
If you have any remarks or complaints regarding this research, you may also
contact the “Research Ethics and Data Management Committee” of REDACTED via
REDACTED.

##  Appendix G Informed consent

By agreeing to this consent form, you confirm that you have read the study
description and that you have been offered the opportunity to ask questions
(via email). Remember that your participation is voluntary, and that you have
the right to decline to participate and withdraw from the research once
participation has begun, without any negative consequences, and without
providing any explanation.

I hereby give permission to:

  * •

Store my anonymised responses to this survey.

  * •

Analyse the anonymised data (both manually and automatically through
statistical software).

  * •

Make the responses to this survey publicly available upon completion of the
study.

Yes $\Rightarrow$ continue to the survey.
No $\Rightarrow$ continue to the end of the survey.

##  Appendix H Task instructions

Task instructions
All questions below are of have the same form. You will see the start of a
sentence on the first line, and a continuation generated by a computer model
on the second line. Your job is to assess the quality of the continuation on
the second line.

Example of a reasonable continuation:
For the following sentence: The clown startled the girl because
A reasonable continuation would be: his make-up was scary.

  * •

Strongly agree

  * •

Somewhat agree

  * •

Neither agree nor disagree

  * •

Somewhat disagree

  * •

Strongly disagree

This is a reasonable continuation because scary make-up can cause someone to
be startled. So here you would answer Strongly agree.

Example of a less reasonable continuation:
For the following sentence: The clown startled the girl because
A reasonable continuation would be: she liked him.

  * •

Strongly agree

  * •

Somewhat agree

  * •

Neither agree nor disagree

  * •

Somewhat disagree

  * •

Strongly disagree

This is a less reasonable continuation, because being liked by someone is
generally not a reason to startle them. So here you would answer one of the
disagree options.