# Partially fake it till you make it:
mixing real and fake thermal images for improved object detection

Francesco Bongini, Lorenzo Berlincioni, Marco Bertini, Alberto Del Bimbo
MICC - Università degli Studi di Firenze Firenze Italy
<EMAIL_ADDRESS>
###### Abstract

In this paper we propose a novel data augmentation approach for visual content
domains that have scarce training datasets, compositing synthetic 3D objects
within real scenes. We show the performance of the proposed system in the
context of object detection in thermal videos, a domain where i) training
datasets are very limited compared to visible spectrum datasets and ii)
creating full realistic synthetic scenes is extremely cumbersome and expensive
due to the difficulty in modeling the thermal properties of the materials of
the scene. We compare different augmentation strategies, including state of
the art approaches obtained through RL techniques, the injection of simulated
data and the employment of a generative model, and study how to best combine
our proposed augmentation with these other techniques. Experimental results
demonstrate the effectiveness of our approach, and our single-modality
detector achieves state-of-the-art results on the FLIR ADAS dataset.

Figure 1: System overview: showing fake+real dataset augmentation, GAN
augmentation, BBox augmentation.

## 1 Introduction

Object detection is a core problem for the perception capabilities of an
autonomous vehicle, the identification of its surroundings and of nearby
objects is essential to ensure a safe deployment of autonomous cars on the
road. In autonomous driving the object detection task is required to be
particularly robust across a range of illumination and environmental
conditions, including daytime, nighttime, rain, fog, etc. In such conditions,
detectors based solely on visible spectrum imagery can easily fail [42, 35],
as shown in Fig. 2.

The use of thermal detectors has recently increased as a mean to mitigate the
sensitivity of visible spectrum imagery to scene-incidental imaging conditions
[35, 29, 4]. A growing number of works have also investigated multispectral
detectors combining visible and thermal images for robust pedestrian detection
[61, 2, 64, 39, 9, 26, 42, 43].

Figure 2: Top row. Left) The lights of the car headlights dazzle the RGB
camera; right) cars and persons can be recognized in the same view obtained
through a thermal camera.
Bottom row. Left) Lack of lighting may cause dangerous situations, no
pedestrian is observable in the scene. Right) the thermal camera does not
depend from the intensity of the light and allows to detect the person.

The cost of multiple aligned sensors however is a limiting factor due to cost,
camera size or availability of lighting to operate the visible spectrum
camera, and multispectral models can have limited applicability in real-world
applications. Moreover, using visible spectrum sensors does not offer the same
privacy-preserving affordances as systems employing only thermal sensors [35].

When compared to multispectral detectors, thermal-only ones typically reach
lower performances, and besides the lack of the information coming from the
visible spectrum there are other challenges for robust object detection using
only thermal data. A key performance-limiting factor is the relative lack of
annotated thermal imagery available for training state-of-the-art models.
Thermal pedestrian datasets are few, and – compared to visible-spectrum
datasets – have orders of magnitude fewer annotated instances. As an example
the Caltech Pedestrian Dataset [18] has 350,000 annotations in the visible
domain, while in thermal domain KAIST Multispectral Pedestrian dataset [33]
has $\sim 51,000$ annotations and FLIR ADAS Dataset [1] has $\sim 28,000$.
Bringing thermal-only detection to the robustness and accuracy demanded by
real-world applications is thus extremely difficult due to the scarcity of
annotated data.

Creating realistic training data from 3D models has become a common practice
in several domains, e.g. for Advanced Driver Assistance Systems (ADAS). An
example is the CARLA simulator [19] that can create RGB, LiDAR, depth, etc.
views of a simulated scene. However, thermal imagery is not yet automatically
generated, since the 3D engine should account for a very large number of
simulation parameters associated to different materials and heat sources.

In this work we propose to simplify the creation of training data adding only
a subset of objects of interest to a real scene, thus reducing the cost of
modeling a whole environment. In order to improve the quality and visual
likelihood of the objects we use a GAN, to further adapt the appearance of the
3D models and to better simulate the output of the shaders used in the 3D
animation engine.

The contributions of this paper are:

  * •

Compositing of 3d fake thermal objects in a real thermal scene: to the best of
our knowledge we are the first to propose this approach, addressing the
thermal spectrum domain, in which creating full virtual scenes is not
practical. We further propose to use a GAN-based approach to adapt the
appearance of the composited objects in the scene.

  * •

Extensive comparison of data augmentation techniques: we provide an extensive
comparison of recent augmentation techniques that have been originally
proposed and evaluated on RGB datasets, like RandAugment [15] and BBAug [70],
on thermal data; we evaluate their combinations, either between them and with
our 3D compositing approach.

  * •

State-of-the-art results: we compare the best thermal augmentations, either
alone or in combination, with a large number of multispectral and thermal
object detectors. Our proposed augmentation technique outperform all the
thermal-only approaches and all the multispectral methods but one.

## 2 Related work

### 2.1 Object detection in thermal imagery

Thanks to the reduction of costs and availability of multispectral cameras
over the past few years, there are numerous recent works exploiting thermal
images in combination with visible images for robust pedestrian detections
[64, 46, 39, 65, 42, 43, 20, 67, 10, 62, 41, 68]. One of the key issues with
thermal imagery is the image resolution and quality is typically far lower
compared to RGB images, and the application of RGB detectors on this low-
quality data yields lower performances. More recently works as MMTOD-UNIT [16]
use a multi-spectral detector that aims to borrow the knowledge from the data-
rich domains such as visual (RGB) without the explicit need for a paired
multimodal dataset. In fact, the model introduces pseudo-visible images
generated from the thermal spectrum ones using the CycleGAN [69], promoting
more information to the training set. The multi-modal Faster-RCNN detector is
then trained. Another recent multi spectral detector is CFRM_3 [66] that
instead cyclically fuse and refine more spectra. The idea is to use a novel
cycle fuse-and-refine module to predict the segmentation mask features of both
visible and thermal spectrum. This allows to complementary get important
features from different spectra. Note that this method uses only aligned
pictures so this is not directly comparable with the other models, although
multispectral cameras typically do not provide aligned frames due to
differences in the focal length of the two lenses. In particular, 4,129 well-
aligned image pairs have been used for training and 1,013 image pairs for
test.

In contrast, many recent works have investigated pedestrian detection in the
thermal (IR) domain only. For example, authors in [34] used Adaptive fuzzy
C-means for IR image segmentation and a CNN for pedestrian detection. In [4]
the authors proposed a combination of Thermal Position Intensity Histogram of
Oriented Gradients (TPIHOG) and the additive kernel SVM (AKSVM) for nighttime-
only detection in thermal imagery. Thermal images augmented with saliency
maps, used as attention mechanism, have been used in [24].

The idea of performing several video preprocessing steps to make thermal
images look more similar to grayscale images converted from RGB was
investigated in [29], who then applied a pretrained and fine-tuned SSD
detector. Recently, authors in [11] designed dual-pass fusion block (DFB) and
channel-wise enhance module (CEM) to improve the one-stage detector RefineDet,
and proposed their ThermalDet detector for pedestrian detection in thermal
imagery. Another recent single-modality work was the Bottom-up Domain
Adaptation approach proposed in [35, 36] for pedestrian detection in thermal
imagery. Task-conditioned training has been recently proposed in [37], adding
the auxiliary task of classifying night and day thermal images and obtaining
state-of-the-art results. Also in this work, we focus on the thermal-only
detection problem.

### 2.2 Data augmentation from 3D models

Creation of virtual images, typically using 3D graphic engines, to improve
object detection and pose estimation in visual spectrum domain has received a
lot of attention from the research community. In the autonomous driving field
in particular countless synthethic datasets and virtual environments [56, 57,
21, 49] are used, especially for reinforcement learning models. The advantages
of using a virtual environment are multiple such as the ease of collection,
the control over the generated data and the possibility to extract information
from multiple, perfectly aligned, virtual sensors i.e. in a 3d videogame the
RGB image, its segmentation map and depth map come for free from the render
program. Specific tools have been developed to ease the creation of these
scenarios, like CARLA [19], that is specialized for the creation of scenarios
for ADAS applications, and VIVID [40], that can be used for indoor navigation,
action recognition and event detection, and provides an advanced human
skeleton system to simulate complex human actions. Most of the effort for
realism in these tools, as well as in the industry of 3d videogame engines, is
focused on the visible spectrum, and the simulated thermal views of a scene
usually lack the nuances of its visible counterpart.

Because of this, works that have explored the use of tools to create augmented
datasets with 3D data have addressed only the visible spectrum context. Some
works have used 3D data to estimate hand [54] or body poses [13, 63]. 3D
avatars have been recently used to enhance human action recognition in [48,
5]. Object detectors have been trained using 3D syntethic data in [7] and
[17]. The former considered classes like person, vehicles and animals in
outdoor city scenes, evaluating the trained detector on surveillance videos.
The latter investigates the effectiveness of rendering engines in generating
realistic scenes for scenarios where no or insufficient annotated data is
available, considering the detection of protective equipment in construction
sites.

To the best of our knowledge we are the first to address compositing of 3D
objects within real scenes in the thermal domain.

### 2.3 Data augmentation from synthetic images

The exploiting of synthetic data from a simulator to obtain a larger, more
diverse, dataset has been explored in many works. This presents a domain
adaptation challenge in which the source domain that needs to be adapted is
partially controllable. One approach to this problem focuses on the model i.e.
by adding to it a domain classifier and using gradient reversal to close the
gap between the two distributions [22] or by freezing the feature extraction
section of the model when training it on synthetic data [36]. Another
technique works on closing the domain shift by having a shared representation
space [31, 58, 47]. Another possible approach instead separately preprocesses
the input data using a learned generative model, trained to perform a
translation between the two domains. [59, 6, 32, 30]. Also in [16] the authors
used a CycleGAN [69] for image-to-image translation of thermal to pseudo-RGB
data. The use of these frameworks to perform data augmentation in order to
improve the performance of a separate classifier has been studied in multiple
previous works such as [3] in which they focus on improving one-shot learning,
in [8] where segmentation of medical images is enhanced by GAN augmented data.
In [55] synthetic data coming from a simulator is adapted and used to train an
RL agent for autonomous driving. Similar to our approach are [30] in which a
much more elaborate cycle consistent framework is developed to perform domain
adaptation and [50] that improves the CycleGAN approach by adding per instance
masks. In [38], a GAN is used to produce fake thermal images to increase the
training data; the authors study the best approach on how to mix these
additional data to real thermal images, showing how adding 10%-20% percent of
fake images improves the performance of the object detection.

### 2.4 Data augmentation from RGB images

AutoAugment [14] is a augmentation framework for vision models that casts the
search of parameters for data augmentation as an optimization problem and
solve it using reinforcement learning. RandAugment [15] improves on
AutoAugment [14] by both considerably reducing the parameters search space
from $10^{32}$ to $10^{2}$ and matching or exceeding performances of [14].
Both this models are tailored to image classification tasks, more recently a
number of works focused on data augmentation specifically developed for
detection problems, such as [53] with a neural rendering approach and [70]
using a similar technique to [15] the authors train a RNN model in a RL
setting to learn optimal augmentation policies including in this
transformations translating, zooming or distorting both the image and the
bounding boxes.

## 3 The proposed method

### 3.1 Mixing real and fake thermal imagery

The Unity game engine [60] is a powerful framework used to develop modern 3D
games. With the help of a 3D artist we exploit this tool to get synthetic data
by rendering sequences of realistic 3D characters and cars superimposed over a
real thermal spectrum image as background. Fig. 3 shows an example of the
process: a real scene from FLIR dataset is used to composite and animate 3D
objects, creating a virtual video sequence. The game engine allowed the use of
a raw thermal shader that simulates the thermal signature of different
objects. The 3D artist needs to concentrate only on creating the raw thermal
appearance of these objects, without need to design the whole scene. This
would have required to compute the thermal properties of all the objects in
the background.

Figure 3: Left) Original thermal image from FLIR dataset [1], Right) Image
with composited fake thermal objects.

We automatically annotated the objects and collected over 10K samples used to
augment the training set. In order to better analyze the synthetic data effect
on the training we produced different sets of simulated scenes changing
parameters such as the source of the background and the number of instances of
the different detected categories (Person, Car, Bicycle).

Figure 4: GAN Input data sample: left) composited 3D fake thermal objects in a
scene; right) segmentation masks of the 3D objects $(img,seg)$

### 3.2 Improving fake thermal imagery appearance through GAN

We use a Generative Adversarial Network to improve the realism of the data
coming from the simulator. In our approach we rely on an LSGAN objective
trained in a cycle-consistent regime as in [69]. In order to achieve this we
trained the generative model in a cycle-consistent setting between the source
domain, the synthetic data, and the target domain, the data from FLIR dataset.
The model architecture is taken from [38] as it proved to be successful in the
thermal images augmentation setting and it is built using the Residual in
Residual Dense Block (RRDB) as the fundamental unit. As in [44], we remove the
batch normalization layer from the traditional Conv-BN-LReLU triplet.

Figure 5: GAN translation synth to real sample (zoomed in and cropped). Left:
Original synthethic image from the simulator; Right: GAN translated version.
The GAN learns to transfer the heat emitted from the bottom of a car, a
feature missing from the simulator.

Similarly to [50] the model is fed both the input image and the segmentation
mask (see Fig. 4) as to provide guidance to the model for the different
thermal signatures of each category. Fig. 5 provides a sample of the synthetic
to real thermal translation; it is interesting to note how the GAN has created
the thermal reflection of the heat of the engine on the concrete of the road,
that was not part of the 3D model. This physical property has been learned by
the GAN from the FLIR thermal dataset. Differently from [50] we do not want
our segmentation mask to be modified by the generative model as this would
change its bounding box and be detrimental to the detector performances.

### 3.3 Other data augmentations

We have also evaluated the applicability of data augmentation techniques
developed for RGB images in the thermal spectrum.

####  3.3.1 RandAugment

We searched the best combinations and strengths of augmentations over a subset
of the FLIR training dataset composed of 5,000 images. However, we noticed
that some of the transformations used are not suitable for the thermal domain,
as shown in Fig. 6.

Figure 6: Examples of augmentations obtained with RandAugment [15]. Left)
translateX and autocontrast have been applied to a picture of the FLIR
dataset; Right) in this case the pedestrians, bicycles and cars are not
detectable at the all due to the solarization transformation.

####  3.3.2 BBox Augmentation

While data augmentation methods like RandAug are often used for image
classification models, Boundary-Box Augmentation [70] (BBAug) focuses on
object detection. In the FLIR dataset the annotations refer to objects like
bikes, pedestrians and cars; the bounding box annotations of these objects
open up the possibility to modify the content of the boxes. The
transformations include, for example, to keep the same image while translating
or zooming the content of the boxes. Starting from an original picture of the
FLIR data set in figure 7, the content of the bounding box in the middle is
modified by this technique.

BBAug [70] is based on the concept of policy, an unordered set of K sub-
policies. Each sub-policy is a set of transformations to be applied to the
image, similarly to the AutoAugment method originally proposed in [14]. The
authors provide the best sub-policies using a Reinforcement Learning search
space algorithm. The models are trained on COCO data set with ResNet-50
backbone [27] and RetinaNet detector [45]. For each picture we want to
augment, a sub-policy is randomly selected. This method allows us to skip the
very expensive search space phase.

Figure 7: Left) An image from FLIR dataset; Right) BBox Augmented version. The
content of the box’s car in the middle is translated downwards and a cutout
square has been added.

### 3.4 Object detector

We used the layer-wise thermal adaptation of YOLOv3 originally proposed in
[35] as object detector, as a baseline for the experiments and to evaluate the
data augmentation technique proposed in this paper.

## 4 Experimental results

### 4.1 Dataset and experimental setup

All of our experiments were conducted on the FLIR ADAS dataset [1]. The
dataset consists of over 10K images, collected using the FLIR Tau2 camera,
both during the day (60%) and the night (40%) driving on Santa Barbara, CA
with clear to overcast weather. We used the three most common classes offered
by the FLIR-ADAS dataset, composed by Person (28,151), Car (46,692), and
Bicycle (4,457), as this is the commonly used experimental setup and allows us
to compare the proposed method with other state-of-the-art approaches.

The training set contains 8,862 frames, while the test set contains 1,366
images taken on different streets from the training set ones. Several
detectors have been compared using the mAP score. For the GAN training we used
a subset of the FLIR-ADAS data from [66] which provides pixel-aligned couples
of RGB and thermal frames. We then used the RGB frames to extract segmentation
masks for pedestrians and vehicles from the scene, using a pretrained on
MSCOCO MaskRCNN [28] from Detectron2 [25].

We devise and test multiple augmentation strategies by combining the different
source of data augmentation previously discussed. From here on we will refer
to BBox Augmentation [70] as “BBAug” and to we will keep the name RandAugment
for the method of [15].

As previously noted, we added synthetic data to the training set. This data is
split into multiple sets in order to perform ablation studies:

  * •

Syntha : Composed of pedestrians walking on a railroad scene; No cars and no
roads. About 4K samples.

  * •

Synthb : Composed of both cars and pedestrians over FLIR-ADAS training set
scenes. About 10K samples.

  * •

Synthc : Composed of both cars and pedestrian over a railroad scene; no roads.
About 4K samples.

We will use the following notation: GAN${{}_{x}}\rightarrow{{}_{y}}$ to refer
to an experiment in which a generative model is trained over the subset
Synthx, performs inference over Synthy and finally this translated Synthy data
is added to the detector training set. We also tested the generative model
from [38] which performs RGB $\rightarrow$ Thermal translation, and refer to
it as GAN${{}_{rgb}}\rightarrow{{}_{th}}$.

### 4.2 Ablation studies

In the first experiment, we investigated several data augmentation strategies
to improve the performances of our detector. In Table 1 we compare the results
of different augmentation techniques and the baseline from [35] in terms of
mean average precision (mAP) for all the detected classes. In the second and
third experiment, we tested RandAugment [15] and BBAug [70], by picking the
best policy for each (see Tab. 2) and the best amount of augmented data (see.
Tab. 3).

Analyzing the results of Tab. 1 we can observe that BBAug [70] performed
better than Randaugment, bringing the base detector mAP to 75%. Employing
Randaugment [15] instead does not show much improvement, probably due to the
fact that its transformations are not developed for object detection as shown
in Fig.6. The combination of the aforementioned does note give better results
than BBAug [70] alone.

The augmentation strategy using the generative model from [38] provides some
improvements compared to the baseline method but only by a 0.3% mAP.

The injection of synthetic data, through compositing of fake 3D objects in a
thermal scene, instead, both directly from the simulator (Syntha, Synthb,
Synthc) and after the domain translation (GAN${{}_{b}}\rightarrow{{}_{a}}$,
GAN${{}_{b}}\rightarrow{{}_{c}}$), raises the detection performances across
all the classes. The best pipeline we found working uses the translated
synthetic data from our Syntha subset, reaching a 76.4% mAP. Also using the
same subset followed by BBAug [70] improved the detector mAP performances to
75.6% which is the second best result. The combination of these two last
techniques does not yield better results. In general, the proposed idea of
compositing fake 3D thermal objects in a thermal scene is an effective
augmentation technique. This is further improved adding BBAug or GAN-based
augmentation that improves the appearance of the 3D objects. Instead,
combining both GAN and BBAug does not improve the results.

Table 1: Comparison of different augmentation strategies in terms of mAP. Best
in bold, second best underlined.

Augmentation Strategy | Person | Bicycle | Car | mAP
---|---|---|---|---
None (Baseline) [35] | 75.6 | 57.4 | 86.5 | 73.2
RandAugment [15] | 74.4 | 60.2 | 85.4 | 73.3
BBAug [70] | 79.4 | 58.4 | 87.2 | 75.0
RandAugment [15] \+ BBAug [70] | 74.6 | 61.6 | 86.0 | 73.9
GAN${{}_{rgb}}\rightarrow{{}_{th}}$ [38] | 77.1 | 56.9 | 86.4 | 73.5
Syntha | 79.3 | 60.1 | 86.2 | 75.2
Syntha \+ BBAug [70] | 77.1 | 64.2 | 85.6 | 75.6
Synthb | 77.2 | 60.0 | 86.2 | 74.5
Synthc | 75.9 | 60.5 | 85.2 | 73.9
GAN${{}_{b}}\rightarrow{{}_{c}}$ | 77.6 | 59.1 | 85.6 | 74.1
GAN${{}_{b}}\rightarrow{{}_{a}}$ | 78.5 | 64.0 | 86.9 | 76.4
GAN${{}_{b}}\rightarrow{{}_{a}}$ \+ BBAug [70] | 74.7 | 64.1 | 86.5 | 75.1

As previously explained, Boundary Box augmentation [70] introduces 4 policies
each composed of 2 possible transformations that can be applied to an image.
We tested all of them and present them in Tab. 2, concluding that the policy
v0 (i.e. TranslateX $p=0.6$, $m=4$, Equalize $p=0.8$, $m=10$) is preferable in
FLIR-ADAS. All the experiments in Tab. 1, Tab. 3, Tab. 4 use this policy.

Table 2: mAP for the 4 policies of the BBAug [70] over a subset of the
synthetic data.

Policies | v0 | v1 | v2 | v3
---|---|---|---|---
Person | 77.1 | 73.4 | 72.2 | 72.9
Bicycle | 64.2 | 53.7 | 54.3 | 54.5
Car | 85.6 | 84.9 | 82.6 | 84.6
mAP | 75.6 | 70.7 | 69.7 | 70.7

We have also explored the effect of different quantities of augmented data
w.r.t the original train set size. In Table 3 a non exhaustive analysis of the
percentage of fake w.r.t real suggests, as previously noted in [38], that the
optimal amount of generated data to add is between 10% and 20%, for the tested
strategies increasing the amount of augmented data becomes detrimental to
learning.

Table 3: Ablation study on varying quantities of augmented images. The
percentage indicates the relative size of the added augmented data.

Technique | @10% | @20% | @50%
---|---|---|---
BBAug [70] | 74.8 | 75.0 | 74.4
RandAugment [15] | 72.9 | 73.3 | 66.5
RandAugment [15] \+ BBAug [70] | 73.9 | 73.7 | 73.6
GAN${{}_{rgb}}\rightarrow{{}_{th}}$ [38] | 73.5 | 72.6 | 72.7
Syntha | 74.2 | 74.1 | 74.0

### 4.3 Comparison with state-of-the-art approaches

In this experiment we compare the best augmentation techniques evaluated in
the previous section, comprising our proposed composition of fake thermal 3D
objects with and without GAN processing, with state-of-the-art approaches. All
the following experiments are evaluated in terms of Mean Average Precision
(mAP).

Table 4: Comparison of the proposed method with state-of-the-art multispectral
and thermal-only methods. Best in bold, second best underlined.

Technique | Spectra | Person | Bicycle | Car | mAP
---|---|---|---|---|---
MMTOD-UNIT [16] | Multi | 64.5 | 49.4 | 70.8 | 61.5
ABiFN [12] | Multi | 66.1 | 48.5 | 71.8 | 62.1
SSD512 VGG16 [51] | Multi | 71.0 | 55.5 | 82.3 | 69.6
CFRM_3 [66] | Multi | 74.5 | 57.8 | 84.9 | 72.4
SSTN101 [52] | Multi | - | - | - | 77.6
YoloV3 Transfer[23] | Thermal | 33.2 | 34.5 | 55.4 | 41.0
SSD VGG-16 Transfer [23] | Thermal | 61.9 | 46.1 | 85.1 | 64.4
Layer-Wise [35] | Thermal | 75.6 | 57.4 | 86.5 | 73.2
RefineDetect512 [1] | Thermal | 79.4 | 58.0 | 85.6 | 74.3
ThermalDet [11] | Thermal | 78.2 | 60.0 | 85.2 | 74.6
BBAug [70] | Thermal | 79.4 | 58.4 | 87.2 | 75.0
Syntha | Thermal | 79.3 | 60.1 | 86.2 | 75.2
Syntha \+ BBAug[70] | Thermal | 77.1 | 64.2 | 85.6 | 75.6
GAN${{}_{b}}\rightarrow{{}_{a}}$ | Thermal | 78.5 | 64.0 | 86.9 | 76.4

Table 4 compares our results with the state-of-the-art single and multi-
spectral approaches. We distinguish multi-spectral detectors, i.e. models
which at test time detect objects using both visible and thermal spectrum
images, and thermal only detectors, i.e. models that use only the thermal
spectrum images. Models which used visible images for transfer learning only
during training fall under the Thermal category.

YoloV3 Transfer[23] and SSD VGG-16 Transfer [23] are both pretrained models on
Coco and ImageNet data sets and tested on FLIR-ADAS. As shown in the Tab.4,
our thermal only detector using different augmentations outperform all but one
of the multi-spectral detectors. Only Self-Supervised Thermal Network
(SSTN101) [52] got better results with 77.57% mAP. Instead, considering
thermal-only detectors, our proposed method outperforms all the competing
approaches.

Fig. 8 compares the detections obtained using the layer-wise YOLOv3 baseline
presented in [36], the results obtained adding the BBAug augmentation
technique [70] and the results obtained using our proposed approach, i.e.
adding 3D models and improving their appearance with a GAN. The examples on
row 2 show that the kid crossing the street was not detected by the method
proposed in [36], but both augmentation approaches are now capable of
correctly detecting him. Row 3 shows how adding 3D models, as proposed in our
method, help to distinguish persons from bicycles, that are not detected with
the two other approaches. Another example of correct detection of a bicycle,
is shown in row 4, were both augmentations are able to recognize it.

Layer-wise - [36] Syntha \- our method BBaug - [70]

Figure 8: Examples of detections: left) Layer-wise adaptation [36], middle)
our method, right) BBaug augmentation. In general, augmentations help to
detect more persons, even in very critical cases like the kid that is crossing
the street on row 2. Our propose method helps the network to better
distinguish bicycle from persons, as shown on row 3.

## 5 Conclusions

In this paper we compared several data augmentation strategies to improve a
YOLOv3 detector working in a thermal-only domain. Given the challenges of the
task, such as its data scarcity, we presented an augmentation pipeline that
combines simulated data within a real scene, followed by domain adaptation
using a generative model. This approach is particularly suitable for domains,
such as thermal imagery, where the creation of completely synthetic scenes is
unfeasible or extremely expensive due to the difficulty of modeling all the
physical properties of the entities of the scene. Our best combination of
strategies reaches states of the art performance in the thermal only domain,
and reaches similar or better results with respect to multi-spectral
detectors.

We gratefully acknowledge the support of NVIDIA Corporation with the donation
of the Titan X Pascal GPU used for this research. This project was partially
funded by Leonardo SpA.

## References

  * fli [2019] 2019\.  URL https://www.flir.com/oem/adas/adas-dataset-form/.
  * Angelova et al. [2015] A. Angelova, A. Krizhevsky, V. Vanhoucke, A. Ogale, and D. Ferguson.  Real-time pedestrian detection with deep network cascades.  In _Proc. of British Machine Vision Conference (BMVC)_ , 2015.
  * Antoniou et al. [2018] A. Antoniou, A. Storkey, and H. Edwards.  Augmenting image classifiers using data augmentation generative adversarial networks.  In _Proc. of Artificial Neural Networks and Machine Learning (ICANN)_ , pages 594–603, 2018.
  * Baek et al. [2017] J. Baek, S. Hong, J. Kim, and E. Kim.  Efficient pedestrian detection at nighttime using a thermal camera.  _Sensors_ , 17(8):1850, 2017.
  * Ballout et al. [2020] M. Ballout, M. Tuqan, D. Asmar, E. Shammas, and G. Sakr.  The benefits of synthetic data for action categorization.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE, 2020.
  * Barth et al. [2020] R. Barth, J. Hemming, and E. Van Henten.  Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation.  _Computers and Electronics in Agriculture_ , 173:105378, 2020.  ISSN 0168-1699.  doi:https://doi.org/10.1016/j.compag.2020.105378.  URL https://www.sciencedirect.com/science/article/pii/S0168169919320794.
  * Bochinski et al. [2016] E. Bochinski, V. Eiselein, and T. Sikora.  Training a convolutional neural network for multi-class object detection using solely virtual world data.  In _Proc. of IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pages 278–285. IEEE, 2016.
  * Bowles et al. [2018] C. Bowles, L. Chen, R. Guerrero, P. Bentley, R. N. Gunn, A. Hammers, D. A. Dickie, M. del C. Valdés Hernández, J. M. Wardlaw, and D. Rueckert.  Gan augmentation: Augmenting training data using generative adversarial networks.  _ArXiv_ , abs/1810.10863, 2018.
  * Brazil et al. [2017] G. Brazil, X. Yin, and X. Liu.  Illuminating pedestrians via simultaneous detection & segmentation.  In _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * Cao et al. [2019] Y. Cao, D. Guan, Y. Wu, J. Yang, Y. Cao, and M. Y. Yang.  Box-level segmentation supervised deep neural networks for accurate and real-time multispectral pedestrian detection.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 150:70–79, 2019.
  * Cao et al. [2019] Y. Cao, T. Zhou, X. Zhu, and Y. Su.  Every feature counts: An improved one-stage detector in thermal imagery.  In _Proc. of IEEE International Conference on Computer and Communications (ICCC)_ , pages 1965–1969, 2019.
  * Charan et al. [2020] A. S. Charan, M. Jitesh, M. Chowdhury, and H. Venkataraman.  Abifn: Attention-based bi-modal fusion network for object detection at night time.  _Electronics Letters_ , 56(24):1309–1311, 2020\.
  * Chen et al. [2016] W. Chen, H. Wang, Y. Li, H. Su, Z. Wang, C. Tu, D. Lischinski, D. Cohen-Or, and B. Chen.  Synthesizing training images for boosting human 3d pose estimation.  In _Proc. of International Conference on 3D Vision (3DV)_ , pages 479–488. IEEE, 2016.
  * Cubuk et al. [2019] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le.  Autoaugment: Learning augmentation strategies from data.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 113–123, 2019.
  * Cubuk et al. [2020] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W)_ , 2020.
  * Devaguptapu et al. [2019] C. Devaguptapu, N. Akolekar, M. M Sharma, and V. N Balasubramanian.  Borrow from anywhere: Pseudo multi-modal object detection in thermal imagery.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W)_ , 2019.
  * Di Benedetto et al. [2020] M. Di Benedetto, F. Carrara, E. Meloni, G. Amato, F. Falchi, and C. Gennaro.  Learning accurate personal protective equipment detection from virtual worlds.  _Multimedia Tools and Applications_ , 2020.  doi:10.1007/s11042-020-09597-9.  URL https://doi.org/10.1007/s11042-020-09597-9.
  * Dollar et al. [2011] P. Dollar, C. Wojek, B. Schiele, and P. Perona.  Pedestrian detection: An evaluation of the state of the art.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 34(4):743–761, 2011.
  * Dosovitskiy et al. [2017] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun.  CARLA: An open urban driving simulator.  In _Proc. of the 1st Annual Conference on Robot Learning_ , pages 1–16, 2017.
  * Fritz et al. [2019] K. Fritz, D. König, U. Klauck, and M. Teutsch.  Generalization ability of region proposal networks for multispectral person detection.  In _Proc. of Automatic Target Recognition XXIX_ , volume 10988, 2019\.
  * Gaidon et al. [2016] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig.  Virtual worlds as proxy for multi-object tracking analysis.  _CoRR_ , abs/1605.06457, 2016.  URL http://arxiv.org/abs/1605.06457.
  * Ganin and Lempitsky [2015] Y. Ganin and V. Lempitsky.  Unsupervised domain adaptation by backpropagation.  In _Proc. of International Conference on Machine Learning (ICML)_ , pages 1180–1189. PMLR, 2015.
  * Gaus et al. [2020] Y. F. A. Gaus, N. Bhowmik, B. K. Isaac-Medina, and T. P. Breckon.  Visible to infrared transfer learning as a paradigm for accessible real-time object detection and classification in infrared imagery.  In _Proc. of Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies IV_ , volume 11542, page 1154205. International Society for Optics and Photonics, 2020.
  * Ghose et al. [2019] D. Ghose, S. M. Desai, S. Bhattacharya, D. Chakraborty, M. Fiterau, and T. Rahman.  Pedestrian detection in thermal images using saliency maps.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W)_ , 2019.
  * Girshick et al. [2018] R. Girshick, I. Radosavovic, G. Gkioxari, P. Dollár, and K. He.  Detectron.  https://github.com/facebookresearch/detectron, 2018.
  * Guan et al. [2019] D. Guan, Y. Cao, J. Yang, Y. Cao, and M. Y. Yang.  Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection.  _Information Fusion_ , 50:148–157, 2019.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask R-CNN.  In _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , pages 2961–2969, 2017.
  * Herrmann et al. [2018] C. Herrmann, M. Ruf, and J. Beyerer.  CNN-based thermal infrared person detection by domain adaptation.  In _Proc. of Autonomous Systems: Sensors, Vehicles, Security, and the Internet of Everything_ , 2018.
  * Hoffman et al. [2018] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. A. Efros, and T. Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In _Proc. of International Conference on Machine Learning (ICML)_ , 2018.
  * Hong et al. [2018] W. Hong, Z. Wang, M. Yang, and J. Yuan.  Conditional generative adversarial network for structured domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1335–1344, 2018.
  * Huang et al. [2018] S.-W. Huang, C.-T. Lin, S.-P. Chen, Y.-Y. Wu, P.-H. Hsu, and S.-H. Lai.  Auggan: Cross domain adaptation with gan-based data augmentation.  In _Proc. of European Conference on Computer Vision (ECCV)_ , pages 718–731, 2018.
  * Hwang et al. [2015] S. Hwang, J. Park, N. Kim, Y. Choi, and I. Kweon.  Multispectral pedestrian detection: Benchmark dataset and baseline.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * John et al. [2015] V. John, S. Mita, Z. Liu, and B. Qi.  Pedestrian detection in thermal images using adaptive fuzzy c-means clustering and convolutional neural networks.  In _Proc. of IAPR International Conference on Machine Vision Applications (MVA)_ , pages 246–249, 2015.
  * Kieu et al. [2019] M. Kieu, A. D. Bagdanov, M. Bertini, and A. Del Bimbo.  Domain adaptation for privacy-preserving pedestrian detection in thermal imagery.  In _Proc. of International Conference on Image Analysis and Processing (ICIAP)_ , 2019.
  * Kieu et al. [2020a] M. Kieu, A. D. Bagdanov, and M. Bertini.  Bottom-up and layer-wise domain adaptation for pedestrian detection in thermal images.  _ACM Transactions on Multimedia Computing Communications and Applications (ACM TOMM)_ , 2020a.
  * Kieu et al. [2020b] M. Kieu, A. D. Bagdanov, M. Bertini, and A. Del Bimbo.  Task-conditioned domain adaptation for pedestrian detection in thermal imagery.  In _Proc. of European Conference on Computer Vision (ECCV)_ , pages 546–562, 2020b.  ISBN 978-3-030-58542-6.
  * Kieu et al. [2021] M. Kieu, L. Berlincioni, L. Galteri, M. Bertini, A. D. Bagdanov, and A. D. Bimbo.  Robust pedestrian detection in thermal imagery using synthesized images, 2021.
  * Konig et al. [2017] D. Konig, M. Adam, C. Jarvers, G. Layher, H. Neumann, and M. Teutsch.  Fully convolutional region proposal networks for multispectral person detection.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W)_ , 2017.  doi:10.1109/CVPRW.2017.36.
  * Lai et al. [2018] K.-T. Lai, C.-C. Lin, C.-Y. Kang, M.-E. Liao, and M.-S. Chen.  Vivid: Virtual environment for visual deep learning.  In _Proc. of ACM Multimedia_ , MM ’18, pages 1356–1359, New York, NY, USA, 2018. Association for Computing Machinery.  ISBN 9781450356657.  doi:10.1145/3240508.3243653.  URL https://doi.org/10.1145/3240508.3243653.
  * Lee et al. [2018] Y. Lee, T. D. Bui, and J. Shin.  Pedestrian detection based on deep fusion network using feature correlation.  In _Proc. of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , 2018.
  * Li et al. [2018] C. Li, D. Song, R. Tong, and M. Tang.  Multispectral pedestrian detection via simultaneous detection and segmentation.  In _Proc. of British Machine Vision Conference (BMVC)_ , 2018.
  * Li et al. [2019] C. Li, D. Song, R. Tong, and M. Tang.  Illumination-aware faster R-CNN for robust multispectral pedestrian detection.  _Pattern Recognition_ , 85:161–171, 2019.
  * Lim et al. [2017] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee.  Enhanced deep residual networks for single image super-resolution.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W)_ , 2017.
  * Lin et al. [2017] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , pages 2980–2988, 2017.
  * Liu et al. [2016] J. Liu, S. Zhang, S. Wang, and D. N. Metaxas.  Multispectral deep neural networks for pedestrian detection.  _arXiv preprint arXiv:1611.02644_ , 2016.
  * Long et al. [2017] M. Long, H. Zhu, J. Wang, and M. I. Jordan.  Deep transfer learning with joint adaptation networks.  In D. Precup and Y. W. Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 2208–2217. PMLR, 06–11 Aug 2017.  URL http://proceedings.mlr.press/v70/long17a.html.
  * Ludl et al. [2020] D. Ludl, T. Gulde, and C. Curio.  Enhancing data-driven algorithms for human pose estimation and action recognition through simulation.  _IEEE Transactions on Intelligent Transportation Systems_ , 21(9):3990–3999, 2020.
  * Martinez et al. [2017] M. Martinez, C. Sitawarin, K. Finch, L. Meincke, A. Yablonski, and A. Kornhauser.  Beyond grand theft auto v for training, testing and enhancing deep learning in self driving cars.  _arXiv preprint arXiv:1712.01397_ , 2017.
  * Mo et al. [2019] S. Mo, M. Cho, and J. Shin.  Instagan: Instance-aware image-to-image translation, 2019.
  * Munir et al. [2020] F. Munir, S. Azam, M. A. Rafique, A. M. Sheri, and M. Jeon.  Thermal object detection using domain adaptation through style consistency.  _arXiv preprint arXiv:2006.00821_ , 2020.
  * Munir et al. [2021] F. Munir, S. Azam, and M. Jeon.  Sstn: Self-supervised domain adaptation thermal object detection for autonomous driving.  _arXiv preprint arXiv:2103.03150_ , 2021.
  * Ning et al. [2021] G. Ning, G. Chen, C. Tan, S. Luo, L. Bo, and H. Huang.  Data augmentation for object detection via differentiable neural rendering.  _arXiv preprint arXiv:2103.02852_ , 2021.
  * Oberweger et al. [2016] M. Oberweger, G. Riegler, P. Wohlhart, and V. Lepetit.  Efficiently creating 3d training data for fine hand pose estimation.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4957–4965, 2016.
  * Pan et al. [2017] X. Pan, Y. You, Z. Wang, and C. Lu.  Virtual to real reinforcement learning for autonomous driving.  _arXiv preprint arXiv:1704.03952_ , 2017.
  * Richter et al. [2016] S. R. Richter, V. Vineet, S. Roth, and V. Koltun.  Playing for data: Ground truth from computer games.  In _Proc. of European Conference on Computer Vision (ECCV)_ , pages 102–118. Springer, 2016.
  * Ros et al. [2016] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3234–3243, 2016.  doi:10.1109/CVPR.2016.352.
  * Sankaranarayanan et al. [2018] S. Sankaranarayanan, Y. Balaji, A. Jain, S. N. Lim, and R. Chellappa.  Learning from synthetic data: Addressing domain shift for semantic segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3752–3761, 2018.
  * Shrivastava et al. [2017] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb.  Learning from simulated and unsupervised images through adversarial training.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2107–2116, 2017.
  * [60] U. Technologies”.  Unity3d,unity development platform.  Web page.  URL https://unity.com/.  Accessed: April 14, 2021.
  * Tian et al. [2015] Y. Tian, P. Luo, X. Wang, and X. Tang.  Pedestrian detection aided by deep learning semantic tasks.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * Vandersteegen et al. [2018] M. Vandersteegen, K. Van Beeck, and T. Goedemé.  Real-time multispectral pedestrian detection with a single-pass deep neural network.  In _Proc. of International Conference Image Analysis and Recognition (ICIAR)_ , 2018.
  * Varol et al. [2017] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev, and C. Schmid.  Learning from synthetic humans.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Wagner et al. [2016] J. Wagner, V. Fischer, M. Herman, and S. Behnke.  Multispectral pedestrian detection using deep fusion convolutional neural networks.  In _Proc. of European Symposium on Artificial Neural Networks (ESANN)_ , 2016.
  * Xu et al. [2017] D. Xu, W. Ouyang, E. Ricci, X. Wang, and N. Sebe.  Learning cross-modal deep representations for robust pedestrian detection.  In _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Zhang et al. [2020] H. Zhang, E. Fromont, S. Lefèvre, and B. Avignon.  Multispectral fusion for object detection with cyclic fuse-and-refine blocks.  In _Proc. of IEEE International Conference on Image Processing (ICIP)_ , pages 276–280. IEEE, 2020.
  * Zhang et al. [2019] L. Zhang, Z. Liu, X. Chen, and X. Yang.  The cross-modality disparity problem in multispectral pedestrian detection.  _arXiv preprint arXiv:1901.02645_ , 2019.
  * Zheng et al. [2019] Y. Zheng, I. H. Izzat, and S. Ziaee.  GFD-SSD: Gated fusion double SSD for multispectral pedestrian detection.  _arXiv preprint arXiv:1903.06999_ , 2019.
  * Zhu et al. [2017] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * Zoph et al. [2020] B. Zoph, E. D. Cubuk, G. Ghiasi, T.-Y. Lin, J. Shlens, and Q. V. Le.  Learning data augmentation strategies for object detection.  In _Proc. of European Conference on Computer Vision (ECCV)_ , pages 566–583. Springer, 2020.


# Representation of Constituents in Neural Language Models:
Coordination Phrase as a Case Study

Aixiu An1 Peng Qian2 Ethan Wilcox3 Roger Levy2
1 Université de Paris, LLF, CNRS<EMAIL_ADDRESS>
2 Department of Brain and Cognitive Sciences, MIT, {pqian<EMAIL_ADDRESS>
3 Department of Linguistics, Harvard University<EMAIL_ADDRESS>
###### Abstract

Neural language models have achieved state-of-the-art performances on many NLP
tasks, and recently have been shown to learn a number of hierarchically-
sensitive syntactic dependencies between individual words. However, equally
important for language processing is the ability to combine words into phrasal
constituents, and use constituent-level features to drive downstream
expectations. Here we investigate neural models’ ability to represent
constituent-level features, using coordinated noun phrases as a case study. We
assess whether different neural language models trained on English and French
represent phrase-level number and gender features, and use those features to
drive downstream expectations. Our results suggest that models use a linear
combination of NP constituent number to drive CoordNP/verb number agreement.
This behavior is highly regular and even sensitive to local syntactic context,
however it differs crucially from observed human behavior. Models have less
success with gender agreement. Models trained on large corpora perform best,
and there is no obvious advantage for models trained using explicit syntactic
supervision.

##  1 Introduction

Humans deploy structure-sensitive expectations to guide processing during
natural language comprehension Levy (2008). While it has been shown that
neural language models show similar structure-sensitivity in their predictions
about upcoming material Linzen et al. (2016); Futrell et al. (2018), previous
work has focused on dependencies that are conditioned by features attached to
a single word, such as subject number Gulordava et al. (2018); Marvin and
Linzen (2018) or wh-question words Wilcox et al. (2018). There has been no
systematic investigation into models’ ability to compute phrase-level
features—features that are attached to a set of words—and whether models can
deploy these more abstract properties to drive downstream expectations.

The star near the moon is $\cdots$ (a)

The star and the moon are $\cdots$ (b)

Figure 1: Subject-verb agreement with (a) the head of a noun phrase structure,
and (b) the coordination structure.

In this work, we assess whether state-of-the-art neural models can compute and
employ phrase-level gender and number features of coordinated subject Noun
Phrases (CoordNPs) with two nouns. Typical syntactic phrases are endocentric:
they are headed by a single child, whose features determine the agreement
requirements for the entire phrase. In Figure 1a, for example, the word _star_
heads the subject NP _The star_ ; since _star_ is singular, the verb must be
singular. CoordNPs lack endocentricity: neither conjunct NP solely determines
the features of the NP as a whole. Instead, these feature values are
determined by compositional rules sensitive to the features of the conjuncts
and the identity of the coordinator. In Figure 1b, because the coordinator is
_and_ , the subject NP number is plural even though both conjuncts (_the star_
and _the moon_) are singular. As this case demonstrates, the agreement
behavior for CoordNPs must be driven by more abstract, constituent-level
representations, and cannot be reduced to features hosted on a single lexical
item.

We use four suites of experiments to assess whether neural models are able to
build up phrase-level representations of CoordNPs on the fly and deploy them
to drive humanlike behavior. First, we present a simple control experiment to
show that models can represent number and gender features of non-coordinate
NPs (Non-coordination Agreement). Second, we show that models modulate their
expectations for downstream verb number based on the CoordNP’s coordinating
conjunction combined with the features of the coordinated nouns (Simple
Coordination). We rule out the possibility that models are using simple
heuristics by designing a set of stimuli where a simple heuristic would fail
due to structural ambiguity (Complex Coordination). The striking success for
all models in this experiment indicates that even neural models with no
explicit hierarchical bias, trained on a relatively small amount of text are
able to learn fine-grained and robust generalizations about the interaction
between CoordNPs and local syntactic context. Finally, we use
subject–auxiliary inversion to test whether an upstream lexical item modulates
model expectation for the phrasal-level features of a downstream CoordNP
(Inverted Coordination). Here, we find that all models are insensitive to the
fine-grained features of this particular syntactic context. Overall, our
results indicate that neural models can learn fine-grained information about
the interaction of Coordinated NPs and local syntactic context, but their
behavior remains unhumanlike in many key respects.

##  2 Methods

###  2.1 Psycholinguistics Paradigm

To determine whether state-of-the-art neural architectures are capable of
learning humanlike CoordNP/verb agreement properties, we adopt the
psycholinguistics paradigm for model assessment. In this paradigm the models
are tested using hand-crafted sentences designed to test underlying network
knowledge. The assumption here is that if a model implicitly learns humanlike
linguistic knowledge during training, its expectations for upcoming words
should qualitatively match human expectations in novel contexts. For example,
Linzen et al. (2016) and Kuncoro et al. (2016) assessed how well neural models
had learned the subject/verb number agreement by feeding them with the prefix
The keys to the cabinet …. If the models predicted the grammatical
continuation are over the ungrammatical continuation is, they can be said to
have learned the number agreement insofar as the number of the head noun and
not the number of the distractor noun, cabinet, drives expectations about the
number of the matrix verb.

If models are able to robustly modulate their expectations based on the
internal components of the CoordNP, this will provide evidence that the
networks are building up a context-sensitive phrase-level representation. We
quantify model expectations as surprisal values. Surprisal is the negative
log-conditional probability $S(x_{i})=-\log_{2}p(x_{i}|x_{1}\dots x_{i-1})$ of
a sentence’s $i^{th}$ word $x_{i}$ given the previous words. Surprisal tells
us how strongly $x_{i}$ is expected in context and is known to correlate with
human processing difficulty (Hale, 2001; Levy, 2008; Smith and Levy, 2013). In
the CoordNP/Verb agreement studies presented here, cases where the proceeding
context sets high expectation for a number-inflected verb form $w_{i}$, (e.g.
singular ‘is’) we would expect $S(w_{i})$ to be lower than its number-
mismatched counterpart (e.g. plural ‘are’).

| Model | Training data | # tokens
---|---|---|---
English | LSTM (PTB) | Penn Treebank |  $\sim$ 1M
ActionLSTM (PTB) | Penn Treebank |  $\sim$ 1M
RNNG (PTB) | Penn Treebank |  $\sim$ 1M
LSTM (enWiki) | English Wikipedia |  $\sim$ 90M
LSTM (1B) | 1 Billion Word |  $\sim$ 800M
French | LSTM (FTB) | French Teebank |  $\sim$ 0.5M
ActionLSTM (FTB) | French Teebank |  $\sim$ 0.5M
RNNG (FTB) | French Teebank |  $\sim$ 0.5M
LSTM (frWaC) | Subset of frWaC |  $\sim$ 138M
Table 1: A summary of models tested.

###  2.2 Models Tested

#### Recurrent Neural Network (RNN) Language Models

are trained to output the probability distribution of the upcoming word given
a context, without explicitly representing the structure of the context
Sundermeyer et al. (2012); Elman (1990). We trained two two-layer recurrent
neural language models with long short-term memory architecture Hochreiter and
Schmidhuber (1997) on a relatively small corpus. The first model, referred as
‘LSTM (PTB)’ in the following sections, was trained on the sentences from Penn
Treebank Marcus and Marcinkiewicz . The second model, referred as ‘LSTM
(FTB)’, was trained on the sentences from French Treebank Abeillé et al.
(2003). We set the size of input word embedding and LSTM hidden layer of both
models as 256.

We also compare LSTM language models trained on large corpora. We incorporate
two pretrained English language models: one trained on the Billion Word
benchmark (referred as ‘LSTM (1B)’) from Jozefowicz et al. (2016), and the
other trained on English Wikipedia (referred as ‘LSTM (enWiki)’) from
Gulordava et al. (2018). For French, we trained a large LSTM language model
(referred as ‘LSTM (frWaC)’) on a random subset (about 4 million sentences,
138 million word tokens) of the frWaC dataset Baroni et al. (2009). We set the
size of the input embeddings and hidden layers to 400 for the LSTM (frWaC)
model since it is trained on a large dataset.

#### ActionLSTM

models the linearized bracketed tree structure of a sentence by learning to
predict the next action required to construct a phrase-structure parse Choe
and Charniak (2016). The action space consists of three possibilities: open a
new non-terminal node and opening bracket; generate a terminal node; and close
a bracket. To compute surprisal values for a given token, we approximate
$P(w_{i}|w_{1\cdots i-1)}$ by marginalizing over the most-likely partial
parses found by word-synchronous beam search Stern et al. (2017).

#### Generative Recurrent Neural Network Grammars (RNNG)

jointly model the word sequence as well as the underlying syntactic structure
Dyer et al. (2016). Following Hale et al. (2018), we estimate surprisal using
word-synchronous beam search Stern et al. (2017). We use the same hyper-
parameter settings as Dyer et al. (2016).

The annotation schemes used to train the syntactically-supervised models
differ slightly between French and English. In the PTB (English) CoordNPs are
flat structures bearing an ‘NP’ label. In FTB (French), CoordNPs are binary-
branching, labeled as NPs, except for the phrasal node dominating the
coordinating conjunction, which is labeled ‘COORD’. We examine the effects of
annotation schemes on model performance in Appendix A. 111The materials and
code for this project can be found in
https://github.com/cpllab/rnn_psycholing_coordination.git

##  3 Experiment 1: Non-coordination Agreement

In order to provide a baseline for following experiments, here we assess
whether the models tested have learned basic representations of number and
gender features for non-coordinated Noun Phrases. We test number agreement in
English and French as well as gender agreement in French. Both English and
French have two grammatical number feature: singular (sg) and plural (pl).
French has two grammatical gender features: masculine (m) and feminine (f).

The experimental materials include sentences where the subject NPs contain a
single noun which can either match with the matrix verb (in the case of number
agreement) or a following predicative adjective (in the case of gender
agreement). Conditions are given in Table 2 and Table 3. We measure model
behavior by computing the plural expectation, or the surprisal of the singular
continuation minus the surprisal of the plural continuation for each condition
and took the average for each condition. We expect a positive plural
expectation in the Npl conditions and a negative plural expectation in the Nsg
conditions. For gender expectation we compute a gender expectation, which is
S(feminine continuation) $-$ S(masculine continuation). We measure surprisal
at the verbs and predicative adjectives themselves.

Condition | Sentence
---|---
Npl | The windows is/are
Nsg | The window is/are
Table 2: Conditions of number agreement in Non-coordination Agreement experiment. Condition | Sentence
---|---
Nm | Les coûts sont importants/importantes
the cost.mpl are important.mpl/fpl
Nf | Les dépenses sont importants/importantes
the expense.fpl are important.mpl/fpl
Table 3: Conditions of gender agreement in Non-coordination Agreement
experiment.

The results for this experiment are in Figure 2, with the plural expectation
and gender expectation on the y-axis and conditions on the x-axis. For this
and subsequent experiments error bars represent 95% confidence intervals for
across-item means. For number agreement, all the models in English and French
show positive plural expectation when the head noun is plural and negative
plural expectation when it is singular. For gender agreement, however, only
the LSTM (frWaC) shows modulation of gender expectation based on the gender of
the head noun. This is most likely due to the lower frequency of predicative
adjectives compared to matrix verbs in the corpus.

(a) English number agreement

(b) French number agreement

(c) French gender agreement

Figure 2: Non-Coordination Agreement experiments for English (number) and
French (number and gender).

##  4 Experiment 2: Simple Coordination

(a) English and-coordination

(b) French and-coordination

(c) English or-coordination

(d) French or-coordination

Figure 3: Comparison of models’ expectation preferences for singular vs.
plural predicate in English and French Simple Coordination experiments.

In this section, we test whether neural language models can use grammatical
features hosted on multiple components of a coordination phrase—the
coordinated nouns as well as the coordinating conjunction—to drive downstream
expectations. We test number agreement in both English and French and gender
agreement in French.

###  4.1 Number Agreement

In simple subject/verb number agreement, the number features of the CoordNP
are determined by the coordinating conjunction and the number features of the
two coordinated NPs. CoordNPs formed by _and_ are plural and thus require
plural verbs; CoordNPs formed by _or_ allow either plural or singular verbs,
often with the number features of the noun linearly closest to the verb
playing a more important role, although this varies cross-linguistically
Fowler et al. (1992). Forced-choice preference experiments in Keung and Staub
(2018) reveal that English native speakers prefer singular agreement when the
closest conjunct in an _or_ -CoordNP is singular and plural agreement when the
closest conjunct is plural. In French, both singular and plural verbs are
possible when two singular NPs are joined via disjunction (Goosse and
Grevisse, 2016).

In order to assess whether the neural models learn the basic CoordNP licensing
for English, we adapted 37 items from Keung and Staub (2018), along the 16
conditions outlined in Table 4. Test items consist of the sentence preamble,
followed by either the singular or plural BE verb, half the time in present
tense (is/are) and half the time in past tense (was/were). We measured the
plural expectation, following the procedure in Section 3. We created 24 items
using the same conditions as the English experiment to test the models trained
in French, using the 3rd person singular and plural form of verb aller, ‘to
go’ (va, vont). Within each item, nouns match in gender; across all conditions
half the nouns are masculine, half feminine.

Condition | Sentence
---|---
pl_and_pl | The doors and the windows is/are
sg_and_pl | The door and the windows is/are
pl_and_sg | The doors and the window is/are
sg_and_sg | The door and the window is/are
pl_or_pl | The doors or the windows is/are
sg_or_pl | The door or the windows is/are
pl_or_sg | The doors or the window is/are
sg_or_sg | The door or the window is/are
Table 4: Conditions of number agreement in Simple Coordination experiment.

The results for this experiment can be seen in Figure 3, with the results for
English on the left and French on the right. The results for and are on the
top row, or on the bottom row. For all figures the y-axis shows the plural
expectation, or the difference in surprisal between the singular condition and
the plural condition. Turning first to English-and (Figure 3a), all models
show plural expectation (the bars are significantly greater than zero) in the
pl_and_pl and sg_and_pl conditions, as expected. For the pl_and_sg condition,
only the LSTM (enWiki) and ActionLSTM are greater than zero, indicating
humanlike behavior. For the sg_and_sg condition, only the LSTM (enWiki) model
shows the correct plural expectation. For the French-and (Figure 3b), all
models show positive plural expectation in all conditions, as expected, except
for the LSTM (FTB) in the sg_and_sg condition.

Examining the results for English-or, we find that all models demonstrate
humanlike expectation in the pl_or_pl and sg_or_pl conditions. The LSTM (1B),
LSTM (PTB), and RNNG models show zero or negative singular expectation for the
pl_or_sg conditions, as expected. However the LSTM (enWiki) and ActionLSTM
models show positive plural expectation in this condition, indicating that
they have not learned the humanlike generalizations. All models show
significantly negative plural expectation in the sg_or_sg condition, as
expected. In the French-or cases, models show almost identical behavior to the
and conditions, except the LSTM (frWaC) shows smaller plural expectation when
singular nouns are linearly proximal to the verb.

These results indicate moderate success at learning coordinate NP agreement,
however this success may be the result of an overly simple heuristic. It
appears that expectation for both plural and masculine continuations are
driven by a linear combination of the two nominal number/gender features
transferred into log-probability space, with the earlier noun mattering less
than the later noun. A model that optimally captures human grammatical
preferences should show no or only slight difference across conditions in the
surprisal differential for the _and_ conditions, and be greater than zero in
all cases. Yet, all the models tested show gradient performance based on the
number of plural conjuncts.

###  4.2 Gender Agreement

In French, if two nouns are coordinated with et (and-coordination), agreement
must be masculine if there is one masculine element in the coordinate
structure. If the nouns are coordinated with ou (or-coordination), both
masculine and feminine agreement is acceptable Corbett (1991); Wechsler and
Zlatić (2003). Although linear proximity effects have been tested for a number
of languages that employ grammatical gender, as in e.g. Slavic languages
(Willer et al., 2018), there is no systematic study for French.

Condition | Sentence
---|---
m_and_m |  6cmLes prix et les coûts sont importants/importantes
the price.mpl and the cost.mpl are important.mpl/fpl
f_and_m |  6cmLes recettes et les coûts sont importants/importantes
the revenues.fpl and the cost.mpl are important.mpl/fpl
m_and_f |  6cmLes prix et les dépenses sont importants/importantes
the price.mpl and the expense.fpl are important.mpl/fpl
f_and_f |  6cmLes recettes et les dépenses sont importants/importantes
the revenues.fpl and the expense.fpl are important.mpl/fpl
Table 5: Conditions for the and-coordination experiment. (Items for or-
coordination are the same except that we change the coordinator to ou.)

To assess whether the French neural models learned humanlike gender agreement,
we created 24 test items, following the examples in Table 5, and measured the
masculine expectation. In our test items, the coordinated subject NP is
followed by a predicative adjective, which either takes on masculine or
feminine gender morphology.

Results from the experiment can be seen in Figure 4. No models shows
qualitative difference based on the coordinator, and only the LSTM (frWaC)
shows significant behavior difference between conditions. Here, we find
positive masculine expectation in the m_and_m and f_and_m conditions, and
negative masculine expectation in the f_and_f condition, as expected. However,
in the m_and_f condition, the masculine expectation is not significantly
different from zero, where we would expect it to be positive. In the or-
coordination conditions, following our expectation, masculine expectation is
positive when both conjuncts are masculine and negative when both are
feminine. For the LSTM (FTB) and ActionLSTM models, the masculine expectation
is positive (although not significantly so) in all conditions, consistent with
results in Section 3.

(a) French and-coordination (b) French or-coordination

Figure 4: Comparison of models’ expectation preferences for Feminine v.s.
Masculine predicative adjectives in French.

##  5 Experiment 3: Complex Coordination

One possible explanation for the results presented in the previous section is
that the models are using a ‘bag of features’ approach to plural and masculine
licensing that is opaque to syntactic context: Following a coordinating
conjunction surrounded by nouns, models simply expect the following verb to be
plural, proportionally to the number of plural nouns.

In this section, we control for this potential confound by conducting two
experiments: In the Complex Coordination Control experiments we assess models’
ability to extend basic CoordNP licensing into sententially-embedded
environments, where the CoordNP can serve as an embedded subject. In the
Complex Coordination Critical experiments, we leverage the sentential
embedding environment to demonstrate that when the CoordNPs cannot plausibly
serve as the subject of the embedded phrase, models are able to suppress the
previously-demonstrated expectations set up by these phrases. These results
demonstrate that models are not following a simple strategy for predicting
downstream number and gender features, but are building up CoordNP
representations on the fly, conditioned on the local syntactic context.

###  5.1 Complex Coordination Control

Following certain sentential-embedding verbs, CoordNPs serve unambiguously as
the subject of the verb’s sentence complement and should trigger number
agreement behavior in the main verb of the embedded clause, similar to the
behavior presented in 4.1. To assess this, we use the 37 test items in English
and 24 items in French in section 4.1, following the conditions in Table 6
(for number agreement), testing only and coordination. For gender agreement,
we use the same test items and conditions for and coordination in Section 4.2,
but with the Coordinated NPs embedded in a context similar to 5.1. As before,
we derived the plural expectation by measuring the difference in surprisal
between the singular and plural continuations and the gender expectation by
computing the difference in surprisal between the masculine and feminine
predicates.

. Je croyais que les prix et les dépenses étaient importants/importantes.
I thought that the.pl price.mpl and the.pl expense.fpl were important.mpl/fpl
I thought that the prices and the expenses were important.

Condition | Sentence
---|---
pl_and_pl | I think that the doors and the windows is/are
sg_and_pl | I think that the door and the windows is/are
pl_and_sg | I think that the doors and the window is/are
sg_and_sg | I think that the door and the window is/are
Table 6: Conditions of number agreement in Complex Coordination Control
experiment.

The results for the control experiments can be seen in Figure 5, with English
number agreement on the top row, French number agreement in the middle row and
French gender agreement on the bottom. The y-axis shows either plural or
masculine expectation, with the various conditions along the x-axis. For
English number agreement, we find that the models behave similarly as they do
for simple coordination contexts. All models show significant plural
expectation when the closest noun is plural, with only two models
demonstrating plural expectation in the sg_and_sg case. The French number
agreement tests show similar results, with all models except LSTM (FTB)
demonstrating significant plural prediction in all cases. Turning to French
gender agreement, only the LSTM (frWaC) shows sensitivity to the various
conditions, with positive masculine expectation in the m_and_m condition and
negative expectation in the f_and_f condition, as expected. These results
indicate that the behavior shown in Section 4.1 extends to more complex
syntactic environments—in this case to sentential embeddings. Interestingly,
for some models, such as the LSTM (1B), behavior is more humanlike when the
CoordNP serves as the subject of an embedded sentence. This may be because the
model, which has a large number of hidden states and may be extra sensitive to
fine-grained syntactic information carried on lexical items Futrell et al.
(2018), is using the complementizer, that, to drive more robust expectations.

(a) English number agreement

(b) French number agreement

(c) French gender agreement

Figure 5: Comparison of model’s expectation preferences in the Complex
Coordination Control experiments.

(a) English number agreement

(b) French number agreement

(c) French gender agreement

Figure 6: Comparison of model’s expectation preferences in the Complex
Coordination Critical experiments.

###  5.2 Complex Coordination Critical

In order to assess whether the models’ strategy for CoordNP/verb number
agreement is sensitive to syntactic context, we contrast the results presented
above to those from a second, critical experiment. Here, two coordinated nouns
follow a verb that cannot take a sentential complement, as in the examples
given in Table 7. Of the two possible continuations—are or is—the plural is
only grammatically licensed when the second of the two conjuncts is plural. In
these cases, the plural continuation may lead to a final sentence where the
first noun serves as the verb’s object and the second introduces a second main
clause coordinated with the first, as in I fixed the doors and the windows are
still broken. For the same reason, the singular-verb continuation is only
licensed when the noun immediately following _and_ is singular.

We created 37 test items in both English and French, and calculated the plural
expectation. If the models were following a simple strategy to drive
CoordNP/verb number agreement, then we should see either no difference in
plural expectation across the four conditions or behavior no different from
the control experiment. If, however, the models are sensitive to the licensing
context, we should see a contrast based solely on the number features of the
second conjunct, where plural expectation is positive when the second conjunct
is plural, and negative otherwise.

Condition | Sentence
---|---
pl_and_pl | I fixed the doors and the windows is/are
sg_and_pl | I fixed the door and the windows is/are
pl_and_sg | I fixed the doors and the window is/are
sg_and_sg | I fixed the door and the window is/are
Table 7: Conditions of number agreement in Complex Coordination Critical
experiment.

Experimental items for a critical gender test were created similarly, as in
Example 7. As with plural agreement, gender expectation should be driven
solely by the second conjunct: For the f_and_m and m_and_m conditions, the
only grammatical continuation is one where the adjectival predicate bears
masculine gender morphology. Conversely, for the m_and_f or f_and_f
conditions, the only grammatical continuation is one where the adjectival
predicate bears feminine morphology. As in 4.1, we created 24 test items and
measured the gender expectation by calculating the difference in surprisal
between the masculine and feminine continuations.

. Nous avons accepté les prix et les dépenses étaient importants/importantes.
we have accepted the.pl price.mpl and the expense.fpl were important.mpl/fpl
We have accepted the prices and the expenses were important.

The results from the critical experiments are in Figure 6, with the English
number agreement on the top row, French number agreement in the middle and
gender expectation on the bottom row. Here the y-axis shows either plural
expectation or masculine expectation, with the various conditions are on the
x-axis. The results here are strikingly different from those in the control
experiments. For number agreement, all models in both languages show strong
plural expectation in conditions where the second noun is plural (blue and
green bars), as they do in the control experiments. Crucially, when the second
noun is singular, the plural expectation is significantly negative for all
models (save for the French LSTM (FTB) pl_and_sg condition). Turning to gender
agreement, only the LSTM (frWaC) model shows differentiation between the four
conditions tested. However, whereas the f_and_m and m_and_f gender
expectations are not significantly different from zero in the control
condition, in the critical condition they pattern with the purely masculine
and purely feminine conditions, indicating that, in this syntactic context,
the model has successfully learned to base gender expectation solely off of
the second noun.

These results are inconsistent with a simple ‘bag of features’ strategy that
is insensitive to local syntactic context. They indicate that both models can
interpret the same string as either a coordinated noun phrase, or as an NP
object and the start of a coordinated VP with the second NP as its subject.

##  6 Experiment 4: Inverted Coordination

In addition to using phrase-level features to drive expectation about
downstream lexical items, human processors can do the inverse—use lexical
features to drive expectations about upcoming syntactic chunks. In this
experiment, we assess whether neural models use number features hosted on a
verb to modulate their expectations for upcoming CoordNPs.

To assess whether neural language models learn inverted coordination rules, we
adapted items from Section 4.1 in both English (37 items) and French (24
items), following the paradigm in Table 8. The first part of the phrase
contains either a plural or singular verb and a plural or singular noun. In
this case, we sample the surprisal for the continuations and (or is
grammatical in all conditions, so it is omitted from this study). Our
expectation is that ‘and’ is less surprising in the Vpl_Nsg condition than in
the Vsg_Nsg condition, where a CoordNP is not licensed by the grammar in
either French or English (as in *What is the pig and the cat eating?). We also
expect lower surprisal for and in the Vpl_Nsg condition, where it is
obligatory for a grammatical continuation, than in the Vpl_Npl condition,
where it is optional.

Condition | Sentence preamble
---|---
Vpl_Npl | What are the doors and
Vpl_Nsg | What are the door and
Vsg_Nsg | What is the door and
Table 8: Conditions in Inverted Coordination experiment.

For French experimental items, the question is embedded into a sentential-
complement taking verb, following Example 6, due to the fact that unembedded
subject-verb inverted questions sound very formal and might be relatively rare
in the training data.

. Je me demande où vont le maire et
I myself ask where go.3PL the.MSG mayor.MSG and

The results for both languages are shown in Figure 7, with the surprisal at
the coordinator on the y-axis and the various conditions on the x-axis. No
model in either language shows a signficant difference in surprisal between
the Vpl_Nsg and Vpl_Npl conditions or between the Vpl_Nsg and Vsg_Nsg
conditions. The LSTM (1B) shows significant difference between the Vpl_Nsg and
Vpl_Npl conditions, but in the opposite direction than expected, with the
coordinator less surprising in the latter condition. These results indicate
that the models are unable to use the fine-grained context sensitivity to
drive expectations for CoordNPs, at least in the inversion setting.

##  7 Discussion

The experiments presented here extend and refine a line of research
investigating what linguistic knowledge is acquired by neural language models.
Previous studies have demonstrated that sequential models trained on a simple
regime of optimizing the next word can learn long-distance syntactic
dependencies in impressive detail. Our results provide complimentary insights,
demonstrating that a range of model architectures trained on a variety of
datasets can learn fine-grained information about the interaction of CoordNPs
and local syntactic context, but their behavior remains unhumanlike in many
key ways. Furthermore, to our best knowledge, this work presents the first
psycholinguistic analysis of neural language models trained on French, a high-
resource language that has so far been under-investigated in this line of
research.

(a) English (b) French

Figure 7: Comparison of models’ surprisals of and-coordination in Inverted
Coordination experiment.

In the simple coordination experiment, we demonstrated that models were able
to capture some of the agreement behaviors of humans, although their
performance deviated in crucial aspects. Whereas human behavior is best
modeled as a ‘percolation’ process, the neural models appear to be using a
linear combination of NP constituent number to drive CoordNP/verb number
agreement, with the second noun weighted more heavily than the first. In these
experiments, supervision afforded by the RNNG and ActionLSTM models did not
translate into more robust or humanlike learning outcomes. The complex
coordination experiments provided evidence that the neural models tested were
not using a simple ‘bag of features’ strategy, but were sensitive to syntactic
context. All models tested were able to interpret material that had similar
surface form in ways that corresponded to two different tree-structural
descriptions, based on local context. The inverted coordination experiment
provided a contrasting example, in which models were unable to modulate
expectations based on subtleties in the syntactic environment.

Across all our experiments, the French models performed consistently better on
subject/verb number agreement than on subject/predicate gender agreement.
Although there are likely more examples of subject/verb number agreement in
the French training data, gender agreement is syntactically mandated and
widespread in French. It remains an open question why all but one of the
models tested were unable to leverage the numerous examples of gender
agreement seen in various contexts during training to drive correct
subject/predicate expectations.

## Acknowledgments

This project is supported by a grant of Labex EFL ANR-10-LABX-0083 (and Idex
ANR-18-IDEX-0001) for AA and MIT–IBM AI Laboratory and the
MIT–SenseTimeAlliance on Artificial Intelligence for RPL. We would like to
thank the anonymous reviewers for their comments and Anne Abeillé for her
advice and feedback.

## References

  * Abeillé et al. (2003) Anne Abeillé, Lionel Clément, and François Toussenel. 2003.  Building a treebank for french.  In _Treebanks_ , pages 165–187. Springer.
  * Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009.  The wacky wide web: a collection of very large linguistically processed web-crawled corpora.  _Language resources and evaluation_ , 43(3):209–226.
  * Choe and Charniak (2016) Do Kook Choe and Eugene Charniak. 2016.  Parsing as language modeling.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2331–2336, Austin, Texas. Association for Computational Linguistics.
  * Corbett (1991) Corbett. 1991.  _Gender_.  Cambridge University Press.
  * Dyer et al. (2016) Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016.  Recurrent neural network grammars.  In _Proceedings of NAACL-HLT_ , pages 199–209.
  * Elman (1990) Jeffrey L Elman. 1990.  Finding structure in time.  _Cognitive science_ , 14(2):179–211.
  * Ficler and Goldberg (2016) Jessica Ficler and Yoav Goldberg. 2016.  Coordination annotation extension in the penn tree bank.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 834–842.
  * Fowler et al. (1992) Henry Ramsey Fowler, Jane E Aaron, and Murray McArthur. 1992.  _The little, brown handbook_.  HarperCollins.
  * Futrell et al. (2018) Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018.  Rnns as psycholinguistic subjects: Syntactic state and grammatical dependency.  _arXiv preprint arXiv:1809.01329_.
  * Goosse and Grevisse (2016) André Goosse and Maurice Grevisse. 2016.  _Le bon usage_.  De Boeck Superieur.
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  _arXiv preprint arXiv:1803.11138_.
  * Hale (2001) John Hale. 2001.  A probabilistic earley parser as a psycholinguistic model.  In _Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies_ , pages 1–8. Association for Computational Linguistics.
  * Hale et al. (2018) John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. 2018.  Finding syntax in human encephalography with beam search.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2727–2736.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016\.  Exploring the limits of language modeling.  _arXiv preprint arXiv:1602.02410_.
  * Keung and Staub (2018) Lap-Ching Keung and Adrian Staub. 2018.  Variable agreement with coordinate subjects is not a form of agreement attraction.  _Journal of Memory and Language_ , 103:1–18.
  * Kuncoro et al. (2016) Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A Smith. 2016.  What do recurrent neural network grammars learn about syntax?  _arXiv preprint arXiv:1611.05774_.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126–1177.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of lstms to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * (20) Mitchell P Marcus and Mary Ann Marcinkiewicz.  Building a large annotated corpus of english: The penn treebank.  _Computational Linguistics_ , 19(2).
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202.
  * Smith and Levy (2013) Nathaniel J Smith and Roger Levy. 2013.  The effect of word predictability on reading time is logarithmic.  _Cognition_ , 128(3):302–319.
  * Stern et al. (2017) Mitchell Stern, Daniel Fried, and Dan Klein. 2017.  Effective inference for generative neural parsing.  _arXiv preprint arXiv:1707.08976_.
  * Sundermeyer et al. (2012) Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012.  Lstm neural networks for language modeling.  In _Thirteenth annual conference of the international speech communication association_.
  * Wechsler and Zlatić (2003) Stephen Wechsler and Larisa Zlatić. 2003.  _The many faces of agreement_.  Stanford: CSLI Publications.
  * Wilcox et al. (2018) Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018.  What do rnn language models learn about filler–gap dependencies?  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 211–221.
  * Willer et al. (2018) J Gold Willer, Boban Arsenijević, Mia Batinić, Michael Becker, Nermina Čordalija, Marijana Kresić, Nedžad Leko, Franc Lanko Marušič, Tanja Milićev, Nataša Milićević, et al. 2018.  When linearity prevails over hierarchy in syntax.  _Proceedings of the National Academy of Sciences of the United States of America_ , 115(3):495–500.

##  Appendix A The Effect of Annotation Schemes

This section further investigates the effects of CoordNP annotation schemes on
the behaviors of structurally-supervised models. We test whether an explicit
COORD phrasal tag improves model performance. We trained two additional RNNG
models on 38,546 sentences from the Penn Treebank annotated with two different
schemes: The first, RNNG (PTB-control) was trained with the original Penn
Treebank annotation. The second, RNNG (PTB-coord), was trained on the same
sentences, but with an extended coordination annotation scheme, meant to
employ the scheme employed in the FTB, adapted from Ficler and Goldberg
(2016). We stripped empty categories from their scheme and only kept the NP-
COORD label for constituents inside a coordination structure. Figure 8
illustrates the detailed annotation differences between two datasets. We
tested both models on all the experiments presented in Sections 3-6 above.

[.NP [.NP [.English ] ] [.and ] [.NP [.comparative ] [.literature ] ] ]

(a) PTB-control

[.NP [.NP-COORD [.English ] ] [.and ] [.NP-COORD [.comparative ] [.literature
] ] ]

(b) PTB-coord

Figure 8: Comparison of annotation schemes of coordination structure.

Turning to the results of these six experiments: We see little difference
between the two models in the Non-coordination agreement experiment. For the
Complex coordination control and Complex coordination critical experiments,
both models are largely the same as well. However, in the Simple and-
coordination and Simple or-coordination experiments the values for all
conditions are shifted upwards for the RNNG PTB-coord model, indicating higher
over-all preference for the plural continuation. Furthermore, the range of
values is reduced in the RNNG PTB-coord model, compared to the RNNG PTB-
control model. These results indicate that adding an explicit COORD phrasal
label does not drastically change model performance: Both models still appear
to be using a linear combination of number features to drive plural vs.
singular expectation. However, the explicit representation has made the
interior of the coordination phrase more opaque to the model (each feature
matters less) and has slightly shifted model preference towards plural
continuations. In this sense, the PTB-coord model may have learned a
generalization about CoordNPs, but this generalization remains unlike the ones
learned by humans.

(a) Non-coordination agreement (b) Simple and-coordination

(c) Simple or-coordination (d) Complex coordination control

(e) Complex coordination critical (f) Inverted coordination

Figure 9: Comparison between RNNGs trained on PTB data with original annotation vs. fine-grained annotation of coordination structure. | PTB | FTB
---|---|---
Condition | sg | pl | total | sg | pl | total
pl_and_pl | 0 | 67 | 67 | 1 | 116 | 116
sg_and_pl | 0 | 72 | 72 | 0 | 50 | 50
pl_and_sg | 0 | 11 | 11 | 0 | 30 | 30
sg_and_sg | 7 | 213 | 220 | 5 | 288 | 293
pl_or_pl | 0 | 2 | 2 | 0 | 14 | 14
sg_or_pl | 0 | 0 | 0 | 0 | 0 | 0
pl_or_sg | 0 | 1 | 1 | 0 | 1 | 1
sg_or_sg | 5 | 1 | 6 | 5 | 8 | 13
Table 9: Frequency of number agreement patterns in PTB and FTB. Condition | m | f | total
---|---|---|---
m_and_m | 38 | 0 | 38
m_and_f | 10 | 1 | 11
f_and_m | 9 | 0 | 9
f_and_f | 0 | 16 | 16
m_or_m | 1 | 0 | 1
m_or_f | 0 | 0 | 0
f_or_m | 1 | 0 | 1
f_or_f | 0 | 1 | 1
Table 10: Frequency of gender agreement patterns in FTB.

##  Appendix B PTB/FTB Agreement Patterns

We present statistics of subject/predicate agreement patterns in the Penn
Treebank (PTB) and French Treebank (FTB) in Table 9 and 10.