# KidSat: satellite imagery to map childhood poverty dataset and benchmark

Makkunda Sharma
Department of Computer Science
University of Oxford
<EMAIL_ADDRESS>
&Fan Yang 11footnotemark: 1
Department of Computer Science
University of Oxford
<EMAIL_ADDRESS>&Duy-Nhat Vo 11footnotemark: 1
Department of Computer Science
University of Oxford
<EMAIL_ADDRESS>
&Esra Suel
The Bartlett Centre for Advanced Spatial Analysis
University College London
<EMAIL_ADDRESS>&Swapnil Mishra
Saw Swee Hock School of Public Health
National University of Singapore
<EMAIL_ADDRESS>&Samir Bhatt
Section of Epidemiology
University of Copenhagen
Department of Infectious Disease Epidemiology
Imperial College London
<EMAIL_ADDRESS>&Oliver Fiala
Save the Children, UK
<EMAIL_ADDRESS>&William Rudgard
Department of Social Policy and Intervention
University of Oxford
<EMAIL_ADDRESS>
&Seth Flaxman
Department of Computer Science
University of Oxford
<EMAIL_ADDRESS>Equal contributionCorrespondence to:
<EMAIL_ADDRESS>
###### Abstract

Satellite imagery has emerged as an important tool to analyse demographic,
health, and development indicators. While various deep learning models have
been built for these tasks, each is specific to a particular problem, with few
standard benchmarks available. We propose a new dataset pairing satellite
imagery and high-quality survey data on child poverty to benchmark satellite
feature representations. Our dataset consists of 33,608 images, each 10 km
$\times$ 10 km, from 19 countries in Eastern and Southern Africa in the time
period 1997-2022. As defined by UNICEF, multidimensional child poverty covers
six dimensions and it can be calculated from the face-to-face Demographic and
Health Surveys (DHS) Program [25]. As part of the benchmark, we test spatial
as well as temporal generalization, by testing on unseen locations, and on
data after the training years. Using our dataset we benchmark multiple models,
from low-level satellite imagery models such as MOSAIKS [20], to deep learning
foundation models, which include both generic vision models such as Self-
Distillation with no Labels (DINOv2) models [15] and specific satellite
imagery models such as SatMAE [6]. We provide open source code for building
the satellite dataset, obtaining ground truth data from DHS and running
various models assessed in our work.

##  1 Introduction

Major satellites like the Landsat or Sentinel program regularly circle the
globe, providing updated, publicly available, high-resolution imagery every
1-2 weeks. An emerging literature in remote sensing and machine learning
points to the promise that these large datasets, combined with deep learning
methods, hold to enable applications in agriculture, health, development, and
disaster response. A cross disciplinary set of publications hint at the
potential impact, showing how satellite imagery can be used to estimate the
causal impact of electrification on livelihoods [18], to measure income,
overcrowding, and environmental deprivation in urban areas [23] and to predict
human population in the absence of census data [27]. Despite these successes,
machine learning for satellite imagery is not yet a well-developed field [19],
with current approaches overlooking the unique features of satellite images
such as variation in spatial resolution over logarithmic scales (from < 1
meter to > 1 km) [19] and the heterogeneous nature of satellite imagery in
terms of the number of bands available from 3 bands for RGB to multispectral
to hyperspectral.

Many areas of machine learning have advanced through the development of
standardized datasets and benchmarks. Given the wide set of possible use cases
for satellite imagery, there is no doubt room for multiple benchmarks. However
there are only a few sources of up-to-date, high-quality satellite imagery,
especially Landsat and Sentinel, so it is natural to construct publicly
available datasets using these satellite programs.

Given the proven effectiveness of remote sensing for tasks that are naturally
visible from space, such as land usage prediction, crop yield forecasting, and
deforestation, we instead choose to focus on a more difficult task:
multidimensional child poverty.

Of the 8 billion people in the world, over 2 billion are children (aged $<$ 18
years old, as defined in the UN Convention on the Rights of the Child [24]).
Child poverty is not the same as adult poverty; children are growing and
developing so they have specific nutrition, health, and education needs—if
these needs are not met, there can be lifelong negative consequences [3].
Poverty cannot simply be assessed by measuring overall household resources, as
households may be very unequal and some of the needs of children, such as
vaccines or education, may be neglected in households that are non-poor.
Instead, child poverty must be measured at the level of the child and their
experience [25].

Child poverty is based on the “constitutive rights of poverty” [25]. What this
concept means is that child poverty includes important dimensions for children
that require material resources to realize them, like education, health and
nutrition, but exclude non-material dimensions such as neglect, violence, and
lack of privacy. Crucially for the purposes of establishing a useful dataset
and benchmark, the internationally agreed definition of child poverty was
designed to enable cross-country comparisons [25].

While other benchmarks exist, most notably SUSTAIN-BENCH [29] which covers a
range of sustainable development indicators, our newly proposed benchmark has
the following features:

  * •

We demonstrate the importance of fine-tuning transformer-based foundation
vision models to tackle a challenging prediction task.

  * •

Child poverty is both a multidimensional outcome, appropriate for fine-tuning
large models, and a univariate measure (percent of children experiencing
severe deprivation ranging from 0% and 100%) meaning that model performance
can be intuitively grasped by policymakers.

  * •

The amount of both satellite and survey data appropriate for child poverty
prediction will continue to increase in the future, as UNICEF is now releasing
geocodes as part of their Multiple Indicator Cluster Survey (MICS) program.

(a) Kriging estimates using Kenya 2022 DHS

(b) DINOv2 fine-tuned on KidSat spatial training dataset

(c) DINOv2 fine-tuned on KidSat temporal training dataset

Figure 1: Estimates of the prevalence of severe deprivation for Kenya in 2022.
(a) shows predictions using a spatial statistics approach, kriging on the
cluster locations using Kenya DHS 2022 data only with a spherical variogram.
(b) shows predictions from DinoV2 fine-tuned on the KidSat spatial dataset, in
which 20% of all clusters in Eastern and Southern Africa were held out. (c)
shows predictions from DinoV2 fine-tuned on the KidSat temporal dataset, in
which the training data was from before 2020.

##  2 Related Work

###  2.1 Existing Satellite Imagery Datasets

With increased access to freely available high resolution satellite imagery
through the Landsat and Sentinel programs, satellite image datasets have
become very popular for training machine learning models. Models and datasets
include functional map of the world (fMoW) [5], XView [14], Spacenet [26], and
Floodnet [17] where the tasks are object detection, instance segmentation, and
semantic segmentation. These are computer vision-specific tasks, rather than
applied health and economic prediction problems, meaning the use of these
datasets and models may be inappropriate for applied health and development
researchers and practitioners.

###  2.2 Satellite Imagery for Demographic and Health Indicators

Machine learning models applied to satellite images are becoming more
commonplace for analysing demographic, health, and development indicators as
they can increase coverage by allowing for interpolation and faster analysis
in under-surveyed regions. In an early work, satellite images were used to
track human development at increasing spatial and temporal granularity [11].
Since then satellite images have been used to track development indicators
which are clearly visible from space such as agriculture and deforestation
patterns [2, 9, 28] but also more abstract quantities such as poverty levels
[1], health indicators [7], and the Human Development Index [22].

###  2.3 Foundation Satellite Image Models

As increasing volumes of data become available, and with progress in self-
supervised learning [10, 4], many foundation models are emerging. In computer
vision, these large models are trained with self-supervised learning on
hundreds of millions of images, serving as a “foundation” from which they can
be fine-tuned for specific tasks. Popular examples of this are Vision
transformers [8], CLIP [16], and DINO [15]. Recently, foundation models have
been trained for satellite imagery specifically on vast amounts of unlabelled
satellite images. Examples of these are SatMAE [6] based on masked
autoencoders, SatCLIP [13] based on CLIP [16], and DiffusionSat [12] which is
a diffusion model [21] for generating satellite images. As it is not yet clear
whether there is a benefit from training foundation models on more specific,
but smaller datasets, we benchmark both generic foundation models for computer
vision as well as satellite-specific foundation models.

##  3 Dataset

In this section, we introduce our unique dataset derived from the Demographic
and Health Surveys (DHS) Program, combining high-resolution satellite imagery
with detailed numerical survey data focused on demographic and health-related
aspects in Eastern and Southern Africa. This dataset leverages the rigorous
survey methodologies from DHS to offer high-quality data on health and
demographic indicators, complemented by satellite images of the surveyed
locations. The rich information embedded in the satellite images enables the
application of advanced deep learning methods to estimate key poverty
indicators in unsurveyed locations.

###  3.1 Satellite Images

This study utilizes high-resolution satellite imagery from two primary
sources: Sentinel-2 and Landsat 5, 7, and 8. These satellite programs are
chosen for their public accessibility, their specific advantages in computer
vision applications, and their long history.

Landsat 5, 7, and 8: Part of a series managed by the United States Geological
Survey (USGS), Landsat 5,7, and 8 together provide imagery with varying
resolutions covering the time span from 1984 to the present (2024).
Specifically, Landsat 8 captures data in 11 bands, including visible, near-
infrared (NIR), and short-wave infrared (SWIR) at 30 meters resolution,
panchromatic at 15 meters, and thermal infrared (TIRS) bands at 100 meters.
This lower resolution for RGB bands contributes to space efficiency in data
storage and processing, making it suitable for large-scale studies over
extensive geographical areas. Additionally, the Landsat program, with missions
dating back to 1972, provides an extensive historical archive of Earth
imagery. This long timespan is particularly advantageous for our study as it
allows the analysis of regions with survey data dating back to 1997.

Sentinel-2: Operated by the European Space Agency (ESA), Sentinel-2 features a
multispectral imager with 13 spectral bands. The resolution varies by band: 10
meters for RGB and NIR, 20 meters for red edge and SWIR bands, and 60 meters
for atmospheric bands. This high resolution in RGB bands provides richer
information, which is valuable for large vision models requiring detailed
visual data for accurate analysis. However, processing such high-resolution
imagery can be computationally expensive, especially when dealing with large
window sizes, posing challenges in terms of processing time and resource
allocation. Additionally, Sentinel-2 only started collecting data in June
2015, which limits its use for analyzing events or changes that occurred
before this date.

For each specified survey coordinate, we extract a 10 km $\times$ 10 km window
of imagery using Google Earth Engine (GEE). Selection criteria for the imagery
include the designation of a specific year and prioritization based on the
least cloud cover within that year. This approach ensures that the images used
are of the highest quality and most suitable for accurate analysis.

Both Sentinel-2 and Landsat series satellites include RGB bands, crucial for
standard object recognition tasks in computer vision. Beyond the RGB spectrum,
these satellites offer additional bands that are instrumental for advanced
remote sensing analysis. This rich assortment of spectral data allows for
sophisticated remote sensing techniques and predictive modeling, such as
estimating vegetation density and water bodies, which are integral to our
study on regional poverty estimation.

###  3.2 Demographic Health Surveys and Child Poverty

Dating back to 1984, the Demographic and Health Surveys (DHS)
Program111http://www.dhsprogram.com has conducted over 400 surveys in 90
countries, funded by the US Agency for International Development (USAID) and
undertaken in partnership with country governments. These nationally
representative cross-sectional household surveys, with very high response
rates, provide up-to-date information on a wide range of demographic, health
and nutrition monitoring indicators. Sample sizes range between 5,000 and
30,000 households, and are collected using a stratified, two-stage cluster
design, with randomly chosen enumeration areas (EAs) called “clusters” forming
the sampling unit for the first stage. In each EA, a random sample of
households is drawn from an updated list of households. DHS routinely collects
geographic information in all surveyed countries. Cluster locations are
released, with random noise added to preserve anonymity with this ’jitter’
being different for rural and urban EAs.

The DHS data include both continuous and categorical variables, each requiring
a different approach for aggregation to ensure accurate ecological analysis at
the cluster level. For continuous variables, we calculated the mean of all
responses associated with a particular spatial coordinate. Min-max scaling was
applied after aggregation to normalise the data, ensuring that all values were
on a scale from 0 to 1. Categorical variables were processed using one-hot
encoding, which converts categories into binary indicator variables.
Similarly, the mean of these binary representations was computed for each
category at each cluster location.

Child poverty was assessed using a methodology formulated by UNICEF that
evaluates child poverty across six dimensions: housing, water, sanitation,
nutrition, health, and education. Each child was classified as moderately or
severely deprived for each dimension based on a set of 17 variables in total
[25]. An overall classification of moderate or severe deprivation is made if
the child experiences moderate or severe deprivation on any of the six
dimensions. Our target quantity of interest, `severe_deprivation`, was
calculated as the percentage of children experiencing severe deprivation
within a cluster. The detailed definition of moderate and severe deprivation
and implementation of poverty calculation can be found in the supplementary
material.

##  4 Benchmark

###  4.1 Spatial

We use 5-fold spatial crossvalidation at the cluster level across countries in
Eastern and Southern Africa, spanning data collected from 1997 to 2022. We
train our models on 80% of the clusters and evaluate its performance using the
mean absolute error (MAE) of the `severe_deprivation` variable on the held-out
20% of clusters. This benchmark is designed to evaluate the model’s capability
to estimate poverty or deprivation levels at any given location based solely
on satellite imagery data, quantifying the model’s generalization capabilities
to unsurveyed locations within surveyed countries.

###  4.2 Temporal

The temporal benchmark employs a historical data training approach, where we
use data collected from 1997 to 2019 as the training set to develop our
models. The objective is to predict poverty in 2020 to 2022. Model performance
is evaluated using the MAE of the `severe_deprivation` variable. This
benchmark tests the model’s ability to capture temporal trends and forecast
poverty based on satellite imagery data, assessing its forecasting accuracy.
This capability is crucial for, e.g. nowcasting poverty before survey data
becomes available.

###  4.3 Models to be Compared

We consider both baseline models (Gaussian process regression, mean
prediction) and a range of more advanced computer vision models, both
unsupervised and semi-supervised, with and without fine-tuning. Each model
represents a distinct strategy in handling and processing satellite imagery:

MOSAIKS [20] is a generalisable feature extraction framework developed for
environmental and socio-economic applications. We obtain MOSAIKS features from
IDinsight, an open-source package that utilizes the Microsoft Planetary
Computer API. The framework leverages satellite imagery to extract meaningful
features from the Earth’s surface. For our purposes, we used its Sentinel
service, querying with specific coordinates, survey year, and a window size of
10 km $\times$ 10 km.

DINOv2 [15] Initially designed for self-supervised learning from images,
DINOv2 excels in generating effective vector representations from RGB bands
alone. For our study, we selected the pre-trained base model with the vision
transformer architecture as the backbone of our foundational model. We fine-
tuned this foundational model with DHS variables to enhance its capability for
predicting poverty. DINOv2 is evaluated in both its raw and fine-tuned forms
using RGB imagery for both spatial and temporal benchmarks.

SatMAE [6] was originally developed for landmark recognition from satellite
imagery. We fine-tuned with DHS variables to enhance its performance for
predicting poverty. SatMAE has 3 variants: RGB, RGB+temporal, and multi-
spectral. For benchmarking, we use the RGB variant for the spatial benchmark,
and RGB+temporal for the temporal benchmark. The RGB+temporal variant takes 3
images of different timestamps from the same location; however, to facilitate
a direct comparison with the other methods which use only a single image, we
provide SatMAE with the same image three times. Additionally, it takes in
Year, Month, and Hour, but since a DHS survey spans up to years, we only
provide the Year variable, with Month and Hour set to January 00:00.

####  4.3.1 Evaluation and Fine-tuning

In our fine-tuning pipeline, we start from DINOv2’s and SatMAE’s original
checkpoints with an uninitialised head and train it against 17 selected DHS
variables to minimize mean absolute error (MAE). We then evaluate the model by
replacing the head with a cross-validated ridge regression model mapping
satellite features to the `severe_deprivation` variable and calculate the MAE
loss on a test set that was neither seen by the fine-tuned model nor the Ridge
Regression. For the spatial task, we perform a 5-fold cross-validation on the
whole dataset, and for the temporal benchmark, we take the training set as the
data before the year 2020 and evaluate on the data from 2020 to 2022.

For the spatial benchmark, we randomly split the data into five train-test
splits using a reproducible script. For the temporal benchmark, we divided the
data into a single fold, using data from before 2020 for training and data
from 2020 onward for testing.

For DINOv2, we used a batch size of 8 for Landsat imagery and a batch size of
1 for Sentinel imagery, with L1 loss and an Adam optimiser of learning rate
and weight decay both set to 1e-6. We trained the model for 20 epochs with
Landsat imagery and 10 epochs with Sentinel imagery, selecting the model with
the minimum validation loss on predicting the 17 DHS variables. Each task was
trained on a single Nvidia V100 32GB GPU, with an average training time of 1
hour per epoch for Landsat and 2 hours per epoch for Sentinel imagery. For
SatMAE, we resize the input to $224\times 224$ and use a batch size of 32 for
the spatial task and 16 for the temporal task. Training is done with Adam
optimiser with learning rate 1e-5 and weight decay 1e-6, for at most 20 epochs
with the early stopping of patience 5 and delta 5e-4. Each task is trained on
a single Nvidia L4 GPU, taking, for Landsat and Sentinel, 1 and 2 hours for
the first epoch and 15 and 10 minutes for each subsequent ones with data
caching.

##  5 Results

The performance of the child poverty prediction models is summarized in Table
1.

###  5.1 Spatial Benchmark

In the spatial benchmarking, Gaussian Process regression with geographic
coordinates resulted in a mean absolute error (MAE) that is 0.04 lower than
that achieved by the baseline mean prediction model. Notably, regressions
using outputs from foundational vision models outperformed both the baseline
and GP regression. The MOSAIKS features based on Sentinel-2 imagery achieved
0.2356 MAE on predicting the `severe_deprivation` variable. Utilising Landsat
imagery, the DINOv2 and SatMAE achieved MAEs of 0.2260 and 0.2341
respectively. Further enhancements through fine-tuning with Demographic and
Health Surveys (DHS) variables led to reduced prediction errors, with DINOv2
and SatMAE recording MAEs of 0.2042 and 0.2125 respectively. When using
Sentinel-2 imagery, the SatMAE architecture achieved errors of 0.2347 and
0.2093 before and after the fine-tuning, while DINOv2 further lowered the
errors to 0.2013 and 0.1836 respectively.

###  5.2 Temporal Benchmark

In the temporal benchmark, models faced greater challenges in forecasting
poverty. Gaussian Process regression was substantially worse than the mean
prediction. Using Sentinel-2 imagery, MOSAIKS recorded an MAE of 0.2588, with
DINOv2 and SatMAE achieving MAEs of 0.2597 and 0.3067 respectively. Additional
fine-tuning with DHS variables led to increased prediction errors, with DINOv2
and SatMAE resulting in MAEs of 0.2858 and 0.3139. Employing Landsat imagery,
the pre-trained DINO v2 and SatMAE model achieved worse initial MAEs of 0.2704
and 0.3453; nevertheless, additional fine-tuning on DHS variables resulted in
relative equal performance for both models, with MAEs of 0.2574 and 0.3376
respectively.

Table 1: Comparison of MAE on severe_deprivation across Benchmarks and Imagery Sources. In the spatial task, random clusters are heldout, while the temporal task is a more difficult forecasting task, with the years 2020-2022 held out. Fine-tuning consistently gives better results. While SatMAE is a foundation model trained on satellite imagery, it is outperformed by the more generic DINOv2 foundation model. Model | Benchmark Type | MAE $\pm$ SE (Spatial) | MAE (Temporal)
---|---|---|---
Mean Prediction | - | $0.2930\pm 0.0018$ | $0.3183$
Gaussian Process Regression | - | $0.2436\pm 0.0002$ | $0.5656$
MOSAIKS | Sentinel-2 | $0.2356\pm 0.0114$ | $0.2588$
DINOv2 (Raw) | LandSat | $0.2260\pm 0.0005$ | $0.2704$
DINOv2 (Raw) | Sentinel-2 | $0.2013\pm 0.0019$ | $0.2597$
DINOv2 (Fine-tuned) | LandSat | $0.2042\pm 0.0015$ | $0.2574$
DINOv2 (Fine-tuned) | Sentinel-2 | $0.1836\pm 0.0036$ | $0.2858$
SatMAE (Raw) | LandSat | $0.2341\pm 0.0017$ | $0.3453$
SatMAE (Raw) | Sentinel-2 | $0.2347\pm 0.0027$ | $0.3067$
SatMAE (Fine-tuned) | LandSat | $0.2125\pm 0.0019$ | $0.3376$
SatMAE (Fine-tuned) | Sentinel-2 | $0.2093\pm 0.0039$ | $0.3139$

###  5.3 Interpretation of Results

The performance of various poverty prediction models is shown in Table 1. Our
prediction task is the percentage of a location’s children who are
experiencing severe deprivation, so a MAE on the order of 0.20 is equivalent
to 20 percentage points of error, which policymakers may consider to be too
high to be useful. The spatial benchmark demonstrates the advantage of using
foundational vision models over the baseline mean prediction model and GPR.
Models like MOSAIKS, DINOv2, and SatMAE, particularly when improved through
fine-tuning with DHS variables, show a further reduction in mean absolute
error (MAE). This implies that spatial features extracted from satellite
imagery are comparably more effective than GP modelling in estimating poverty
indicators in regions where surveys have not been conducted.

The temporal benchmark, which evaluated a forecasting task (predict 2020-2022
using data from before 2020), was more difficult than the spatial benchmark.
Satellite imagery is at best a proxy for multidimensional child poverty, and
this finding suggests it is a better proxy for quantifying spatial as opposed
to temporal variation. Satellite imagery models performed worse on the
temporal as compared to spatial benchmark, and the fine-tuned models,
particularly those using Sentinel-2 imagery as the source input, showed
increased MAE compared to the raw output from both DINOv2 and SatMAE models.
This suggests that the models overfit the historical data, and struggled to
generalise to data collected after 2020. Gaussian process regression based on
spatial coordinates had no way of predicting changes over time, explaining its
very poor performance.

##  6 Discussion and Future Work

###  6.1 Satellite Imagery Sources

As compared to Landsat, models utilising Sentinel-2 imagery, such as the fine-
tuned versions of DINOv2 and MOSAIKS, demonstrated improved performance in
both spatial benchmarks. These models benefited from the rich spectral
information provided by Sentinel-2, which enabled more precise predictions of
deprivation levels across diverse geographical regions.

Additionally, the computational demands associated with processing high-
resolution Sentinel-2 data present substantial challenges. For instance, large
versions of vision transformers could not be accommodated within the memory
constraints of a 32 GB GPU when processing the full Sentinel-2 data. In
contrast, these larger models could be deployed with Landsat data, which
offers lower resolution but requires less computational resources. Under the
spatial setting, this scenario highlights a critical trade-off in model
deployment: the choice between employing lightweight models to retain the high
resolution of Sentinel-2 imagery or opting for more powerful models that
necessitate a reduction in image resolution to ensure feasibility.

###  6.2 Modeling Choices

We considered a representative set of models: MOSAIKS is unsupervised, DINOv2
is a generic foundation model trained on images, and SatMAE is a foundation
model trained on satellite imagery.

####  6.2.1 MOSAIKS

MOSAIKS is designed to provide general-purpose satellite encodings and is
notably accessible through Microsoft’s Planetary Computer service. This model
generates a large output vector, typically around 4000 dimensions, which,
while comprehensive, can lead to significant computational costs when methods
beyond simple linear regression are employed. Furthermore, although MOSAIKS is
well-suited for broad applications, integrating online feature acquisition
into a fine-tuning process tailored specifically to poverty prediction
presents challenges. This limitation can hinder its effectiveness when
adapting to specific tasks where dynamic feature updates are crucial. We also
note that MOSIAKS’ API at times returned no-features, even after implementing
rate-limiting mechanisms. This random behaviour combined with unavailability
of features before 2013 limits the use of MOSAIKS considerably.

####  6.2.2 DINOv2

DINOv2 stands out as a state-of-the-art foundational model that excels in
generating effective vector representations from RGB bands alone, achieving
comparable performance to models that utilize additional spectral bands. Its
flexibility in model sizing allows users to select the optimal model scale for
specific training needs, enhancing its utility across various computational
settings. The availability of pre-trained weights simplifies the process of
fine-tuning for specialized tasks such as poverty prediction. However,
DINOv2’s reliance solely on RGB bands means it does not leverage the broader
spectral information available in other satellite imagery bands, which may
limit its application scope to scenarios where such data could provide
additional predictive insights.

####  6.2.3 SatMAE

SatMAE demonstrates respectable results, surpassing baseline models even with
only its raw, pre-trained configuration. Its architecture inherently supports
the integration of multispectral and temporal analysis, making it well-suited
for handling complex datasets typically encountered in satellite imagery
analysis. Despite these strengths, the pre-trained SatMAE model is configured
to process images of 224 $\times$ 224 pixels, constraining its ability to
utilize higher-resolution imagery, such as the 1000 $\times$ 1000 pixel images
from Sentinel-2. This limitation restricts its performance, particularly in
comparison to models that can fully exploit high-resolution data, thereby
failing to match the effectiveness of other advanced models in our analysis.
Another limitation is that with our simple benchmarking setup, we have not
made full use of SatMAE’s temporal and multi-spectral capabilities. In the
temporal setup, we are providing only one image-timestamp pair with only the
Year variable, while the model is capable of taking up to 3 pairs, along with
Month and Hour variables. We are also exclusively using the RGB bands, while
the multi-spectral version of SatMAE is capable of taking in other bands of
the satellite images in our dataset.

###  6.3 Further Discussion

The ability to accurately measure poverty across a vast number of geolocations
is crucial for understanding and addressing the disparities that exist in
different regions. The extensive and high-quality poverty measurement is
valuable for researchers and policymakers. It allows for the analysis of
poverty trends and the effectiveness of current policies, thereby facilitating
more informed decision-making to reduce global poverty.

Traditional surveys, while rich in data, are limited by geographical and
logistical constraints. Conducting extensive on-the-ground surveys is not only
costly but also time-consuming—from data collection to processing and
harmonisation. In regions lacking detailed survey data, traditional methods
like GPR or nearest-neighbor approaches are typically used to estimate poverty
levels. However, these methods can be unreliable, particularly when
extrapolating data to locations far from surveyed areas or data with temporal
dependencies, leading to high uncertainty.

On the other hand, satellite imagery, which was made widely available by
organizations such as the European Space Agency and the United States
Geological Survey, can be accessed from any geographic location. Recent
advancements in the field of computer vision have made it possible to infer
meaningful information from this imagery, which can effectively improve
poverty prediction. By demonstrating the capabilities of large vision models
and satellite imagery in this context, we aim to inspire and encourage others
in the field to further develop and refine these methods, thus driving changes
in sociology research and policy making.

###  6.4 Limitation and Future Directions

Our study had a number of limitations. While high-quality household survey
data is expensive to acquire, it is an irreplaceable source of ground truth;
machine learning can complement and enhance, but never replace, these
datasets. We highlighted the difficulty of the temporal benchmark, suggesting
that future research could explore time series methods for forecasting or ways
of better encoding temporal information into the foundation models. Another
limitation of our study is that, in our goal of learning general
representations of satellite imagery, we fine-tuned the large vision models to
predict DHS variables important to calculating the `severe_deprivation`
variable rather than directly optimising the models on the
`severe_deprivation` variable itself. While this approach could lead to better
generalisation by leveraging a broader range of demographic and health
indicators, it may not be the most effective strategy for minimising the MAE
specifically for `severe_deprivation`. This indirect optimisation can result
in sub-optimal performance for the target variable. Future work could explore
the extent of benchmark improvement with direct optimisation techniques for
`severe_deprivation`. Additionally, future work could also evaluate the
performance of various models on the six components of child poverty
separately, on moderate as opposed to severe deprivation. While we evaluated
spatial generalisation by leaving out clusters, a stricter evaluation would
have considered a leave-one-country out evaluation.

##  7 Conclusion

In conclusion, our study demonstrates the potential of satellite imagery
combined with large vision models to estimate child poverty across spatial and
temporal settings. We introduced a new dataset pairing publicly accessible
satellite images with detailed survey and child poverty data based on the
Demographic and Health Surveys Program, covering 19 countries in Eastern and
Southern Africa over the period 1997-2022. By benchmarking multiple models,
including foundational vision models like MOSAIKS, DINOv2, and SatMAE, we
assessed their performance in predicting child poverty. Our results show that
advanced models with satellite imagery have the potential to outperform
baseline methods, offering more accurate and generalisable poverty estimates.
This work highlights the importance of integrating remote sensing data with
machine learning techniques to address complex socio-economic issues,
providing a scalable and cost-effective approach for poverty estimation and
policy-making.

##  8 Acknowledgments

We thank Matthew Sutcliffe for his help during the early stages of this work,
and Kaustubh Chakradeo, Enrique Delamónica, and Lucie Cluver for productive
discussions. We acknowledge the support of Google Cloud for Researchers. S.F.
and M.S. acknowledge the Alan Turing Institute. S.F. acknowledges the EPSRC
(EP/V002910/2). S.M. acknowledges support from the National Research
Foundation via The NRF Fellowship Class of 2023 Award (NRF-NRFF15-2023-0010).
S.B. acknowledges funding from the MRC Centre for Global Infectious Disease
Analysis (reference MR/X020258/1), funded by the UK Medical Research Council
(MRC). This UK funded award is carried out in the frame of the Global Health
EDCTP3 Joint Undertaking. S.B. is funded by the National Institute for Health
and Care Research (NIHR) Health Protection Research Unit in Modelling and
Health Economics, a partnership between UK Health Security Agency, Imperial
College London and LSHTM (grant code NIHR200908). Disclaimer: “The views
expressed are those of the author(s) and not necessarily those of the NIHR, UK
Health Security Agency or the Department of Health and Social Care.”. S.B.
acknowledges support from the Novo Nordisk Foundation via The Novo Nordisk
Young Investigator Award (NNF20OC0059309). SB acknowledges the Danish National
Research Foundation (DNRF160) through the chair grant. S.B. acknowledges
support from The Eric and Wendy Schmidt Fund For Strategic Innovation via the
Schmidt Polymath Award (G-22-63345).

## References

  * [1] K. Ayush, B. Uzkent, K. Tanmay, M. Burke, D. Lobell, and S. Ermon.  Efficient poverty mapping from high resolution remote sensing images.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12–20, 2021.
  * [2] J. G. Ball, K. Petrova, D. A. Coomes, and S. Flaxman.  Using deep convolutional neural networks to forecast spatial patterns of amazonian deforestation.  Methods in Ecology and Evolution, 13(11):2622–2634, 2022.
  * [3] J. Brooks-Gunn and G. J. Duncan.  The effects of poverty on children.  The future of children, pages 55–71, 1997.
  * [4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [5] G. Christie, N. Fendley, J. Wilson, and R. Mukherjee.  Functional map of the world.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172–6180, 2018.
  * [6] Y. Cong, S. Khanna, C. Meng, P. Liu, E. Rozi, Y. He, M. Burke, D. Lobell, and S. Ermon.  Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery.  Advances in Neural Information Processing Systems, 35:197–211, 2022.
  * [7] A. Daoud, F. Jordán, M. Sharma, F. Johansson, D. Dubhashi, S. Paul, and S. Banerjee.  Using satellite images and deep learning to measure health and living standards in india.  Social Indicators Research, 167(1):475–505, 2023.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [9] L. D. Estes, S. Ye, L. Song, B. Luo, J. R. Eastman, Z. Meng, Q. Zhang, D. McRitchie, S. R. Debats, J. Muhando, et al.  High resolution, annual maps of field boundaries for smallholder-dominated croplands at national scales.  Frontiers in artificial intelligence, 4:744863, 2022.
  * [10] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  * [11] N. Jean, M. Burke, M. Xie, W. M. Davis, D. B. Lobell, and S. Ermon.  Combining satellite imagery and machine learning to predict poverty.  Science, 353(6301):790–794, 2016.
  * [12] S. Khanna, P. Liu, L. Zhou, C. Meng, R. Rombach, M. Burke, D. Lobell, and S. Ermon.  Diffusionsat: A generative foundation model for satellite imagery.  arXiv preprint arXiv:2312.03606, 2023.
  * [13] K. Klemmer, E. Rolf, C. Robinson, L. Mackey, and M. Rußwurm.  Satclip: Global, general-purpose location embeddings with satellite imagery.  arXiv preprint arXiv:2311.17179, 2023.
  * [14] D. Lam, R. Kuzma, K. McGee, S. Dooley, M. Laielli, M. Klaric, Y. Bulatov, and B. McCord.  xview: Objects in context in overhead imagery.  arXiv preprint arXiv:1802.07856, 2018.
  * [15] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [16] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [17] M. Rahnemoonfar, T. Chowdhury, A. Sarkar, D. Varshney, M. Yari, and R. R. Murphy.  Floodnet: A high resolution aerial imagery dataset for post flood scene understanding.  IEEE Access, 9:89644–89654, 2021.
  * [18] N. Ratledge, G. Cadamuro, B. De la Cuesta, M. Stigler, and M. Burke.  Using satellite imagery and machine learning to estimate the livelihood impact of electricity access.  Technical report, National Bureau of Economic Research, 2021.
  * [19] E. Rolf, K. Klemmer, C. Robinson, and H. Kerner.  Mission critical–satellite data is a distinct modality in machine learning.  arXiv preprint arXiv:2402.01444, 2024.
  * [20] E. Rolf, J. Proctor, T. Carleton, I. Bolliger, V. Shankar, M. Ishihara, B. Recht, and S. Hsiang.  A generalizable and accessible approach to machine learning with global satellite imagery.  Nature communications, 12(1):4392, 2021.
  * [21] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [22] L. Sherman, J. Proctor, H. Druckenmiller, H. Tapia, and S. M. Hsiang.  Global high-resolution estimates of the united nations human development index using satellite imagery and machine-learning.  Technical report, National Bureau of Economic Research, 2023.
  * [23] E. Suel, S. Bhatt, M. Brauer, S. Flaxman, and M. Ezzati.  Multimodal deep learning from satellite and street-level imagery for measuring income, overcrowding, and environmental deprivation in urban areas.  Remote Sensing of Environment, 257:112339, 2021.
  * [24] UN General Assembly.  Convention on the rights of the child.  United Nations, Treaty Series, 1577(3):1–23, 1989.
  * [25] UNICEF Data.  Child poverty profiles: Understanding internationally comparable estimates, 2021.
  * [26] A. Van Etten, D. Lindenbaum, and T. M. Bacastow.  Spacenet: A remote sensing dataset and challenge series.  arXiv preprint arXiv:1807.01232, 2018.
  * [27] N. Wardrop, W. Jochem, T. Bird, H. Chamberlain, D. Clarke, D. Kerr, L. Bengtsson, S. Juran, V. Seaman, and A. Tatem.  Spatially disaggregated population estimates in the absence of national population and housing census data.  Proceedings of the National Academy of Sciences, 115(14):3529–3537, 2018.
  * [28] J. Xu, A. Elmustafa, L. Weldegebriel, E. Negash, R. Lee, C. Meng, S. Ermon, and D. Lobell.  Harvestnet: A dataset for detecting smallholder farming activity using harvest piles and remote sensing.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22438–22446, 2024.
  * [29] C. Yeh, C. Meng, S. Wang, A. Driscoll, E. Rozi, P. Liu, J. Lee, M. Burke, D. B. Lobell, and S. Ermon.  Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning.  arXiv preprint arXiv:2111.04724, 2021.

##  Appendix A Appendix / supplemental material

###  A.1 Dataset description

The KidSat dataset we present in this work includes both cluster-wise child
poverty derived from the DHS data and the satellite imagery corresponding to
each cluster. Due to the confidentiality of the survey data, DHS requires
registration prior to accessing the data. We include detailed procedures for
acquiring the satellite imagery and DHS data in our GitHub repository.

####  A.1.1 Coding child poverty

The `severe_deprivation` variable is used in this work to represent severe
child poverty for individual responses within the cluster. It is calculated by
aggregating several indicators of severe deprivation across multiple
dimensions such as housing, water, sanitation, nutrition, health, and
education. A child is considered severely deprived if they experience at least
one severe deprivation in any of the following areas:

  1. 1.

Housing: Severe deprivation if the number of persons per room is 5 or more.

  2. 2.

Water: Severe deprivation if the household uses unsafe water sources.

  3. 3.

Sanitation: Severe deprivation if the household lacks access to safe
sanitation facilities.

  4. 4.

Nutrition: Severe deprivation if a child’s height-for-age z-score is below -3
(indicating severe stunting).

  5. 5.

Health: Severe deprivation if a child misses all essential vaccines or has
untreated acute respiratory infections.

  6. 6.

Education: Severe deprivation if a school-aged child does not attend school
and has not completed any level of education.

Among all DHS variables involved in child poverty calculation, we selected 17
variables, presented in Table 2, as the prediction objective during model
fine-tuning. We scaled the continuous variables to the range from 0 to 1. We
expanded the categorical columns using one-hot encoding, where each column is
in the format of `variable_value`. In total, a vector of dimension 99 based on
these 17 DHS variables was used to represent a cluster, where we mapped the
satellite imagery to predict these vectors as the method of model fine-tuning.

Table 2: DHS variables selected for model fine-tuning Variable |  Description |  Type
---|---|---
h10 |  Whether the child ever received any vaccination to prevent diseases. |  Categorical
h3 |  DPT 1 vaccination. |  Categorical
h31 |  Whether the child had suffered from a cough in the last two weeks and whether the child had been ill with the cough in the last 24 hours. |  Categorical
h5 |  DPT 2 vaccination. |  Categorical
h7 |  DPT 3 vaccination. |  Categorical
h9 |  Measles 1 vaccination. |  Categorical
hc70 |  Height for age standard deviation (according to WHO). |  Continuous
hv106 |  Highest level of education the household member attended. |  Categorical
hv109 |  Educational attainment recoded. |  Categorical
hv121 |  Household member attended school during current school year. |  Categorical
hv201 |  Main source of drinking water for members of the household. |  Categorical
hv204 |  Time taken to get to the water source for drinking water |  Continuous
hv205 |  Type of toilet facility in the household. |  Categorical
hv216 |  Number of rooms used for sleeping in the household. |  Continuous
hv225 |  Whether the household shares a toilet with other households. |  Categorical
hv271 |  Wealth index factor score (5 decimals) |  Continuous
v312 |  Current contraceptive method. |  Categorical

###  A.2 Compute

As one of the heavy-lifting parts is loading images, a multi-core CPU ($\geq
8$) is recommended to optimise the data loading using multiple workers with
the data loader. The training was done using Nvidia Tesla V100 GPUs for DINO
v2 experiments and Nvidia L4 GPUs for SatMAE experiments. In particular, for
DINO v2 experiments with Sentinel imagery, 32 GB of GPU memory is a hard
requirement.

###  A.3 Experimental details

The dataset for our experiments was sourced from Sentinel-2 and Landsat-8
satellite imagery. Each image tile, corresponding to specific geographical
centroids, was preprocessed by selecting and stacking RGB bands, normalising,
and clipping pixel values to the 0-255 range. Data was organised into cross-
validation folds with separate training and testing sets. Only centroids with
available imagery were included, and rows with missing target values were
excluded to maintain data integrity. (Note: As an unsupervised method, no
fine-tuning was attempted for MOSAIKS; see main text for details.)

####  A.3.1 DINOv2

We used the base DINOv2 Vision Transformer (ViT) model which has an output of
768-dimension feature vector. We appended a regression head for mapping the
feature vector to the 99-dimension target DHS variables. The Adam optimiser
was used, with a learning rate and weight decay of 1e-6 for both the base
model and the regression head. Training involved iterating over 20 epochs for
Landsat imagery and 10 epochs for Sentinel imagery to minimise the L1 loss
between predicted and actual target values. The batch size was determined by
available GPU memory, 8 for Landsat imagery source and 1 for Sentinel. The
average training time is 1 hour per epoch for Landsat experiments and 2 hours
per epoch for Sentinel experiments.

####  A.3.2 SatMAE

The base SatMAE model is the decoder of a MAE, outputting a 1024-dimensional
vector. Fine-tuning is done by appending a transformer head mapping to the
99-dimensional target DHS variables. We use the Adam optimiser, with a
learning rate of 1e-5 and weight decay of 1e-6 for both the base model and the
head. Training involved iterating over 20 epochs for both Landsat imagery and
Sentinel imagery to minimise the L1 loss between predicted and actual target
values, with early stopping of patience 5 and delta 5e-4. The batch size is 32
for spatial task and 16 for temporal task. Training takes 1 hour for the first
epoch and 15-30 minutes for each subsequent one when data caching is used.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: We only make factual claims about the dataset and benchmark in the abstract.

  5. 2.

Limitations

  6. Question: Does the paper discuss the limitations of the work performed by the authors?

  7. Answer: [Yes]

  8. Justification: There is a section on limitations.

  9. 3.

Theory Assumptions and Proofs

  10. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  11. Answer: [N/A]

  12. Justification: The paper does not contain theoretical results.

  13. 4.

Experimental Result Reproducibility

  14. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  15. Answer: [Yes] ,

  16. Justification: We are releasing all the code and full instructions for benchmarking and dataset creation.

  17. 5.

Open access to data and code

  18. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  19. Answer: [Yes]

  20. Justification: We are releasing all the code and full instructions for benchmarking and dataset creation.

  21. 6.

Experimental Setting/Details

  22. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  23. Answer: [Yes]

  24. Justification: Training details and experimental setting included in the paper. Full code also being released.

  25. 7.

Experiment Statistical Significance

  26. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  27. Answer: [Yes]

  28. Justification: we report standard errors from 5-fold cross validation, under the assumption of approximate normality.

  29. 8.

Experiments Compute Resources

  30. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  31. Answer: [Yes]

  32. Justification: The appendix will include a section on compute resources.

  33. 9.

Code Of Ethics

  34. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  35. Answer: [Yes]

  36. 10.

Broader Impacts

  37. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  38. Answer: [Yes]

  39. Justification: We discuss the potential impact our work has, as it can allow for finer grained and faster analysis of child poverty, a very important social issue.

  40. 11.

Safeguards

  41. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  42. Answer: [N/A]

  43. Justification: The dataset released does not fall in the above categories.

  44. 12.

Licenses for existing assets

  45. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  46. Answer: [Yes]

  47. Justification:

     * •

Landsat images are in the open domain.

     * •

Sentinel images have been released under the Open Access Creative Commons CC
BY-SA 3.0 IGO licence.

     * •

The DHS data will be used by every user after signing a terms of use agreement
with them.

  48. 13.

New Assets

  49. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  50. Answer: [Yes]

  51. Justification: We are releasing all the code and full instructions for benchmarking and dataset creation. Details will be included in appendix.

  52. 14.

Crowdsourcing and Research with Human Subjects

  53. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  54. Answer: [Yes]

  55. Justification: The data used in this paper is provided by the Demographic and Health Surveys Program. The methodology can be found on DHS’s website. The link is documented in the supplementary code repository.

  56. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  57. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  58. Answer: [N/A]

  59. Justification: [N/A]


# Structural Ambiguity and its Disambiguation in Language Model Based Parsers:
the Case of Dutch Clause Relativization

Gijs Wijnholds
Institute of Language Sciences
Utrecht University
<EMAIL_ADDRESS>
Michael Moortgat
Institute of Language Sciences
Utrecht University
<EMAIL_ADDRESS>

###### Abstract

This paper addresses structural ambiguity in Dutch relative clauses. By
investigating the task of _disambiguation by grounding_ , we study how the
presence of a prior sentence can resolve relative clause ambiguities. We apply
this method to two parsing architectures in an attempt to demystify the
parsing and language model components of two present-day neural parsers.
Results show that a neurosymbolic parser, based on proof nets, is more open to
data bias correction than an approach based on universal dependencies,
although both setups suffer from a comparable initial data bias.

##  1 Introduction

Ambiguity pervades natural language and as such forms one of the central
challenges for natural language understanding (NLU) systems. Given the fact
that most such systems rely on large-scale deep learning architectures, the
presence of structural biases in the training data used may affect a system’s
capacity for disambiguation. Specifically in the case of parsing, typical
architectures rely on the assumption of just a single correct parse, although
many may exist. This assumption then may force a bias into the training, both
on the lexical and on the syntactic level.

In this paper, we study syntactic ambiguities in Dutch, where a structural
ambiguity affects the interpretation of relative clauses. The preferred
reading as subject or object relativisation will typically be determined by
lexical choice. A running example in Dutch, with its two possible
interpretations in English, is given below:

$(a)$ | de dokter die de patiënt geneest
---|---
$(b)$ | _the doctor who cured the patient_
$(c)$ | _the doctor whom the patient cured_

In this example, the verb ‘cure’ displays a strong selectional preference for
a doctor as its subject and a patient as its object, so one would expect that
the subject-relative interpretation $(b)$ is the preferred reading. However,
grounding the phrase in a prior sentence disambiguates the ambiguous relative
clause: by prepending the phrase “The patient cured the doctor”, one is led to
infer that the object-relative reading of $(c)$ is in fact preferred.
Orthogonally, there are cases of lexical choice that _block_ one of the
readings, typically due to the semantic class of the subject and object being
different. An example is “De man die de boterham eet” (_The man who eats the
sandwich_), where it is semantically implausible for the verb’s arguments to
be reversed. The situation is summarized in Table 1, where different possible
orderings of the relative clause together with a prior sentence lead to a
different expected readings of the relative clause.

Arguing that a parser typically exploits statistical properties of its
training corpus, but should additionally rely on both lexical and syntactic
cues inside of that corpus, we carry out an experiment test a parser’s
capacity for disambiguation in context. Specifically, we extract a set of
selectional preferences for Dutch transitive verbs, that we classify according
to their _reversibility_ , i.e. whether subject and object can be
interchanged, additionally indicating whether there is a strong preference for
a noun as subject or object. This leads to three classes of $(s,v,o)$ triples
that we then use to generate a test set of Dutch relative clauses together
with prior sentences, to test a parser’s capability of disambiguation in
context.

By evaluating two different parsing regimes that both are built on top of a
language model, we investigate the encoding of structural bias in the parser
training data, the possibility of mitigating structural bias, and attempt to
pinpoint to what extent lexical knowledge or syntactic information is employed
by the parsers in question.

Prior sentence | Target phrase | Correct reading | Plausible
---|---|---|---
De man eet de boterham. | De man die de boterham eet | subj. rel. | ✓
De man eet de boterham. | De boterham die de man eet | obj. rel. | ✓
De boterham eet de man. | De man die de boterham eet | n/a | ✗
De dokter geneest de patient. | De dokter die de patient geneest | subj. rel. | ✓
De patient geneest de dokter. | De dokter die de patient geneest | obj. rel. | ✓
De dokter geneest de patient. | De patient die de dokter geneest | obj. rel. | ✓
De patient geneest de dokter. | De patient die de dokter geneest | subj. rel. | ✓
Table 1: Different cases in our disambiguation experiment, where the provided
prior sentence determines the interpretation of the target phrase. The top
three rows are cases of an _irreversible_ $(s,v,o)$ triple where interchanging
subject and object leads to an implausible case that is not included in our
experiments. The bottom four rows are cases of a _reversible_ $(s,v,o)$,
albeit with a strong selectional preference for one interpretation. By adding
the prior sentence, the interpretation of the target phrase is disambiguated.

Our contributions in this paper are therefore threefold: we (1) provide a
dataset of selectional preferences for Dutch $(s,v,o)$ triples with an
additional layer of classification according to semantic noun classes, and (2)
create a novel test set targeting structural ambiguity in the interpretation
of Dutch relative clauses. Finally, we (3) provide a number of experiments
indicating that structural bias is easily encoded, but not so easily mitigated
in a language model-based parser.

##  2 Background

#### Probing and syntactic sensitivity

Previous work has used probing, where a small neural network is attached to a
large language model to extract task-specific information, to argue that large
scale language models like BERT have internalized some linguistic knowledge
during pretraining Tenney et al. (2019), and there appears to be some
consensus of the syntactic awareness of BERT models Rogers et al. (2020).
Specifically, studies have indicated the possibility of extracting parse trees
from BERT representations succesfully Hewitt and Manning (2019); Vilares et
al. (2020).

Another line of research into syntactic sensitivity investigates the
probabilities of language models in a masked language modelling environment,
where studies typically define surprisal rates to measure the degree to which
the language model’s predictions coincide with human-like behaviour in the
face of syntactic ambiguities Futrell et al. (2019); Hu et al. (2020);
Arehalli et al. (2022); Aina and Linzen (2021), typically focusing on garden-
path effects. A related approach uses priming to investigate the language
models’ response to structurally similar sentences Sinclair et al. (2022).

These studies differ from the current paper in that we explicitly target
sentences that are not disambiguated without adding extrasentential context,
which should for a parser to infer the intended syntactic analysis. Hence, our
setup relies on a probing-like paradigm where we evaluate a parser on top of a
language model.

#### Dutch NLP

The rising interest in large scale language models in the NLP community has
led to a number of investigations for Dutch specifically. Two dominant Dutch
language models have been developed, based on the respective BERT Devlin et
al. (2019) (BERTje, de Vries et al. (2019)) and the RoBERTa Liu et al. (2019)
architecture (RobBERT, Delobelle et al. (2020)). On the evaluation side there
have been several studies using Dutch-specific phenomena to evaluate the
respective monolingual language models: the work of Wijnholds and Moortgat
(2021) introduces a parallel Natural Language Inference (NLI) dataset for
Dutch, showing that Dutch NLI is more difficult to tackle than its original
English version. More recently, a range of probing studies has been performed
investigating verb-subject dependencies in the face of syntactic constructions
involving discontinuity Kogkalidis and Wijnholds (2022) and ellipsis Haagen et
al. (2022), aside from a more subtle Dutch NLI challenge Wijnholds (2023).

More closely related to constructions involving relative pronouns is the work
of Allein et al. (2020), which introduces relative pronoun prediction for
Dutch _die_ and _dat_ as a binary classification task, where the surrounding
context determines the choice of the neuter (_dat_) or non-neuter (_die_)
pronoun. This can be modelled as a masked language modelling task Delobelle et
al. (2020), reaching high performance. A more complex variation of this task
is defined by Bouma (2021), where the experiment investigates the language
model’s capacity to predict relative pronoun attachment.

Against these studies, our approach differs in that we do not directly target
the language model probabilities, but rather investigate the attached parsers,
thereby indirectly assessing the contextualization power of the underlying
language model. In that sense, the current work is more in line with prior,
more theoretical work that takes parser ambiguity into account Moortgat and
Wijnholds (2017); Moortgat et al. (2020).

Finally, while previous work tried to exploit selectional preferences to
improve parser accuracy van Noord (2010), we rather use selectional preference
to investigate the bias of existing parsers, as a precursor to potential
architectural considerations, in a setting where language models are commonly
employed in parser development. We make the code for the different components
available in three separate
repositories.111https://github.com/gijswijnholds/syntactic_nl2i
222https://github.com/gijswijnholds/dynamic-proof-nets-
disambiguation333https://github.com/gijswijnholds/udparsing_bert

##  3 Data Generation

In order to generate a suitably large set of relative clause patterns, we
proceed with a pipeline for extracting suitable subject-verb-object triples
from a large corpus. First we perform a probability-based extraction of base
triples. Then we use lexical information from a dictionary to filter out
relevant nouns and classify them according to their semantic class, after
which in a final step we perform a manual filtering and classification of the
obtained subject-verb-object triples to guarantee correctness.

#### Triple extraction

For the first step, we need a way to extract subject-verb-object triples from
a large corpus. To this end, we iterate over Lassy Large van Noord et al.
(2013) – a 700M word corpus with automatically assigned syntactic annotations
– and extract all the cases of transitive verbs and their respective subjects
and objects.

To compensate for parsing errors and infrequent observations, we rely on
posterior probability Resnik (1997), the simplest measure that was shown to
give best performance on a selectional preference acquisition task Zhang et
al. (2019). Posterior probability allows us to easily filter out those triples
that occurred most frequently, allowing us to consider the most canonical
triples only. For a given $(s,v,o)$ triple, its posterior probability is
defined as follows:

$p(s,v,o)=\frac{f(s,v,o)}{\sum\limits_{s^{\prime},o^{\prime}}f(s^{\prime},v,o^{\prime})}$

#### Dictionary-based classification

After gathering all $(s,v,o)$ triples and removing stopwords, we perform a
dictionary-based filtering and classification in two steps. First, we obtain
an exhaustive list of Dutch nouns with their respective semantic class444
_person, animal, plant, substance, object, abstract, mass noun_ from the
_Algemeen Nederlands Woordenboek_ 555https://anw.ivdnt.org, a comprehensive
online dictionary of Dutch.

We use these categories to classify all the subjects and objects in the
extracted $(s,v,o)$ triples, after which we organize triples according to the
most frequent noun categories, preventing as much as possible infrequent or
illicit combinations of a verb with a given pair of nouns. By organizing
triples by whether subject and object fall into the same semantic category, we
have an initial estimate of whether a triple is _irreversible_ , and in cases
where the semantic category of the subject and object coincide, we estimate
how strong the preference of the verb for the particular subject and object is
based on the frequency of the $(s,v,o)$ triple relative to its inverted
$(o,v,s)$ triple.

#### Manual filtering

This initial estimation gives us a comprehensive list of subject-verb-object
triples, that we finally filter manually, making a per case decision on
whether a triple is _irreversible_ , reversible with _strong_ preference for
the given subject and object, or reversible with a _weak_ such preference,
meaning that object and subject could be swapped without leading to an
implausible triple. An example of the latter is “De toerist herkent de
reiziger” (_The tourist recognizes the traveller_).

#### Generating relative clauses

After the two-step process described above, we have a robust set of $(s,v,o)$
triples that we can use to generate the desired test cases for our experiment.
In total, we obtained 3304 irreversible triples, 370 triples with a strong
preference for the regular relation, and 724 where subject and object could be
easily interchanged.

From these triples, we generate relative clauses following the pattern
displayed in Table 1. First, for an irreversible $(s,v,o)$ triple, we create a
relative clause with the subject $s$ as the head noun (S Pron O V) as well as
the reversed relative clause (O Pron S V), which due to the irreversibility of
the $(s,v,o)$ triple must be analyzed with the subject-relative and object-
relative reading respectively. We prepend in both cases a prior sentence in
SVO order to be able to inspect the effect of adding this context to the
parser’s input.

For the reversible triples, we generate four cases. Again, we generate two
variations of the relative clause, but additionally vary the prior sentence so
it will force a reading of the relative clause, that is ambiguous without this
context. This allows us to compare parser performance on both lexical and
syntactic cues.

In the generation process, we use the _Algemeen Nederlands Woordenboek_ to
extract the gender of each noun (_de_ for gendered nouns, _het_ for neuter
nouns), and the gender of the relative pronoun (_die_ for gendered head nouns,
_dat_ for neuter head nouns).

##  4 Parsing Regimes

In our experiments we want to evaluate parsers that were built on top of a
large language model, in order to distinguish the effect of the language model
from that of the specific parsing strategy employed. For comparison purposes
we test two different parsing regimes.

#### Neural proof nets

The first parser we examine is a neurosymbolic parser based on a multi-modal
type-logical grammar that simultaneously encodes function-argument structure
and dependency roles Kogkalidis et al. (2023, 2020). This parsing setup exists
alongside other _neuralizations_ of categorial grammar parsers Clark (2021),
but was explicitly developed for Dutch.

The architecture of this parser follows the typical structure of a categorial
parser, where a _supertagging_ component assigns to words logical formulas
that encode their intended combinatorial behaviour, followed by a process of
_proof search_ that combines the formulas into a proof representing the full
parse history. In the system of Kogkalidis et al. (2023), supertagging is
implemented as a graph decoding network, that learns to construct the tree
structures representing the formulas of the logical formalism from an
underlying (Dutch) BERT model de Vries et al. (2019). Proof search is
implemented as _neural proof net_ search, which amounts to linking atomic
subformulas of opposite polarity in a way that determines the correct
dependency and function-argument relations.

For a full exposition of this parser we refer the reader to Kogkalidis et al.
(2020); for the sake of our experiments it is enough to consider the two
possible correct supertagging assignments for the Dutch relative clause,
illustrated in Table 2.

#### Universal dependencies

The second parser we evaluate approaches parsing as a sequence labelling task,
following the work of Strzyz et al. (2019).

| De | patiënt | die | de | dokter | geneest
---|---|---|---|---|---|---
Subj. rel. | $\Box_{det}(\textsf{N}\multimap\textsf{NP})$ | N | $\lozenge_{relcl}(\lozenge_{su}\textsf{VNW}\multimap\textsf{S})\multimap\Box_{mod}(\textsf{NP}\multimap\textsf{NP})$ | $\Box_{det}(\textsf{N}\multimap\textsf{NP})$ | N | $\lozenge_{obj1}\textsf{NP}\multimap\lozenge_{su}\textsf{VNW}\multimap\textsf{S}$
Obj. rel. | $\Box_{det}(\textsf{N}\multimap\textsf{NP})$ | N | $\lozenge_{relcl}(\lozenge_{obj1}\textsf{VNW}\multimap\textsf{S})\multimap\Box_{mod}(\textsf{NP}\multimap\textsf{NP})$ | $\Box_{det}(\textsf{N}\multimap\textsf{NP})$ | N | $\lozenge_{obj1}\textsf{NP}\multimap\lozenge_{su}\textsf{VNW}\multimap\textsf{S}$
Table 2: Supertagging assignment of the neural proof net parser for the subject-relative and object-relative interpretation of the Dutch relative clause. Function-argument structure is encoded by the linear implication, where $A\multimap B$ denotes a function consuming a phrase of type $A$ to produce a result of type $B$. The unary operations $\lozenge_{d},\Box_{d}$ encode dependency structure, where heads assign dependency role $d$ to their _complements_ by means of $\lozenge_{d}$ marking, and $\Box_{d}$ allows _adjuncts_ to project their dependency role $d$. The parser assigns a _higher-order_ formula to the relative pronoun, allowing the implicit gap for either the subject or object of the verb (“geneest”) in the body of the relative clause to be identified with the head noun (“de patiënt”). For the sake or our experiments, we can inspect the formula assigned to the relative pronoun ‘die’ to determine the interpretation of the relative clause, given the annotation of the gap type VNW with either the $su$ or $obj1$ dependency. | De | patiënt | die | de | dokter | geneest
---|---|---|---|---|---|---
Subj. rel. | (+1,N,$det$) | (-1,ROOT, $root$) |  (+1,V,$nsubj$) | (+1,N,$det$) | (+1,V,$obj$) | (-2,N,$acl$:$relcl$)
Obj. rel | (+1,N,$det$) | (-1,ROOT, $root$) |  (+1,V,$obj$) | (+1,N,$det$) | (+1,V,$nsubj$) | (-2,N,$acl$:$relcl$)
Table 3: Relative part-of-speech based encoding of the subject-relative and
object-relative interpretation of the Dutch relative clause. In both cases,
the label assigned to the relative pronoun ‘die’ determines the interpretation
of the relative clause.

Specifically, sentences are encoded using a relative part-of-speech based
encoding, with each word assigned a triple $(i,p,d)$ where $p$ refers to the
part-of-speech of the word’s head, $i$ indicating its relative location, and
$d$ the word’s dependency label. For example, the label (+1, V, $nsubj$) says
that the current word is in the _nsubj_ dependency relation with respect to
the first word to its right that carries the V part-of-speech tag. The root is
encoded by labelling its dependent with (-1, ROOT, $root$). A full example is
given in Table 3 which contains the encoding of the two possible parses for
the Dutch relative clause. The relative part-of-speech based encoding was
found to be the highest performing in the experiments of Strzyz et al. (2019)
and so we use it in our experiments. Labels are considered atomically, and as
such the parser is implemented as a standard token classification model, where
a token-level classifier is fine-tuned along with a BERT model.

##  5 Evaluation & Results

In our experiments, we test the parsers on three different scenarios: first,
we inspect the existing structural bias present in the parsers due to training
data statistics, in a setting where the parser only gets fed the relative
clause. In such a setting, we would expect the parser to have a strong bias
for irreversible cases, but an even distribution in accuracy on weakly
reversible cases. Next, we observe the effect of parsing the relative clause
when the underlying language model is allowed to contextualize against the
prior sentence, expecting this to aid the parser in assigning the correct
reading. In this setting, we would ideally hope for high accuracy across the
board. Finally, we examine the effect of additionally finetuning the parser
components on a small amount of training data to see if the parser can pick up
on the task.

###  5.1 Experimental setup

We organize the test cases into a train/dev/test split in order to compare the
parser baseline against a finetuning setting, where the parser is additionally
trained to recognize examples of disambiguating context to learn how to choose
the correct interpretation of the relative clause.

#### Data preparation

In order to not allow overfitting of the model – we feed it data that is
engineered to be task-specific – we select a small amount of training data
against larger development and test sets. We separate the verbs involved in
the three different data sets as another measure to avoid overfitting. This
leads to a training set of 2640 samples, against a development set of 4400
samples and a test set of 5556 samples.

#### Parser setup

We initially train both parsers on the original examples from Lassy-Small, as
the neural proof net parser was trained on this. However, plain evaluation on
the test data is not an option given that we want to prepend a prior
disambiguating sentence; the fact that BERT employs positional embeddings
makes the parsers unsuitable for the task. Hence we train the parsers from
scratch, prepending a random number of unattended tokens, ranging between 5-80
tokens. This ensures that the parser will be robust against the position
shifting in later experiments. Due to the difference in parsing regime,
baseline scores for the position-shifted parsers follow different evaluation
metrics, displayed in Table 4.

_NPN_ |  | _UD_
---|---|---
92.98 | 54.87 |  | 88.37 | 86.92
Table 4: Baseline parser accuracy scores.

For the neural proof net parser we report tagging accuracy (percentage of
total supertags correctly predicted) and frame accuracy (percentage of
sentences for which all supertags were correctly predicted). These numbers are
only slightly lower than the original parser.666Tagging accuracy: 93.21, frame
accuracy: 56.36 For the UD parser, we compute unlabelled and labelled
attachment score, which are comparable to state of the art.777The Spacy Dutch
UD parser nl_core_news_lg reports a UAS score of 87 and a LAS score of 83
(https://spacy.io/models/nl)

###  5.2 First experiment

For the initial evaluation of the parsers, we assess their disambiguation
performance in two scenarios: first, we ask the parsers to parse only the
ambiguous phrase, inspecting the initial bias the parsers have obtained from
their training data. In the next experiment, we allow the underlying language
model to contextualize against the disambiguating prior sentence, and we let
the parser component then parse the ambiguous phrase to see if it can
succesfully exploit the LM’s contextualization capabilities.

Table 5 displays the results for the first scenario, where we present the
relative clause in two possible orders: one in which the regular order is
presented (S Pron O V), and one in which subject and object are interchanged
(O Pron S V). In the case of irreversible triples, this means that in the
regular order we expect a high-performing parser to always assign the subject-
relative interpretation, but for the reversed order we expect the object-
relative interpretation. For the reversible cases, we expect a 50/50 accuracy
for both presentations if there is a weak preference for either order, and a
skew towards the subject-relative interpretation in the case of strong lexical
preference.

_Neural Proof Nets_ | S die O V | O die S V
---|---|---
| (_subj-rel_) | (_obj-rel_)
Irreversible | 98.76 | 61.86
Reversible-strong | 95.37 | 2.32
Reversible-weak | 97.64 | 1.63
_Universal Dependencies_ | S die O V | O die S V
| (_subj-rel_) | (_obj-rel_)
Irreversible | 95.72 | 27.85
Reversible, Strong pref. | 94.88 | 0.65
Reversible, Weak pref. | 92.03 | 0.26
Table 5: Accuracy results for three different relative clauses without
context. Left: presenting the relative clause in regular word order. Right:
presenting the relative clause in reversed order. These results indicate the
baseline parsing preference without any disambiguating prior sentence.

The result displays a clear preference for the subject-relative
interpretation. In the case of irreversible triples the parsers both pick up
on the fact that presented a reversed order must obtain the object-relative
interpretation. On the other hand, the results for the reversible triples are
significantly below expectation, in the sense that regardless of the presented
word order, they will almost always assign a subject-relative interpretation
($>95\%$ on the left, $<6\%$ on the right).

These results are somewhat to be expected: the subject-relative this reading
prevails in the training data that the parsers were trained on: a total of 306
cases of the subject-relative interpretation occur in Lassy Small, versus 32
cases of the object-relative interpretation. One could then argue that this is
indeed the natural intended interpretation so it should have been picked up by
any parser replicating its training data, explaining the results in Table 5.

#### Contextualization

However, if it were the case that the parser can easily exploit the
information embedded in the language model, we would expect to see that
setting the model such that the prior sentence is indeed attended to by the
underlying BERT model, the performance would increase. Table 6 displays the
results for this second scenario. Here, by introducing the prior sentence as
disambiguating context, the ideal parser scores upward to $100\%$ everywhere,
thus indicating it can make the correct parse in context.

First | SVO | SVO | OVS | OVS
---|---|---|---|---
Second | S die O V | O die S V | S die O V | O die S V
Reading | (_subj-rel_) | (_obj-rel_) | (_obj-rel_) | (_subj-rel_)
_NPN_ |  |  |  |
Irrev. | 98.88 | 69.77 | N/A | N/A
Strong | 98.98 | 10.13 | 1.81 | 93.05
Weak | 98.83 | 4.76 | 2.14 | 97.19
_UD_ |  |  |  |
Irrev. | 96.91 | 35.03 | N/A | N/A
Strong | 96.61 | 1.29 | 0.51 | 96.06
Weak | 94.77 | 0.46 | 0.45 | 94.30
Table 6: Accuracy results for three different relative clauses with context,
i.e. the prior sentence is attended to by the language model prior to parsing.
These results indicate the effect of contextualizing on parsing disambiguation
performance.

We observe a stable accuracy for the cases of subject-relative readings, with
significant accuracy gains for the object-relative reading that is less
persistent in the original training data. This shows that, to some extent, the
added information of the underlying language model gives the parser the
incentive to pick up on the grammatical relations in the prior sentence.
However, the results are not particularly encouraging: while both parsers do
increase in accuracy overall, their strong bias towards a subject-relative
interpretation remains.

###  5.3 Finetuning

After evaluation of the influence of the prior sentence through the
contextualization of the BERT embeddings, we additionally finetune the parsers
on our task, to see whether the parser could in principle assign the correct
interpretation given that the underlying BERT embeddings have access to the
prior sentence for contextualization. We explicitly do not update the language
model itself, as we want to investigate the parser’s capacity for
disambiguation, and allowing the language model to update would be too prone
to overfitting Rogers et al. (2020).

The finetuning scenario thus serves as a means to measure the extent to which
the parsers’ strong structural bias can be mitigated. Given the fact that the
relative clause has an unambiguous interpretation in the contextualized
scenario, training is straightforward, and the results reflect whether the
parser by itself can pick up on the contextualized lexical information
provided by the language model. These results are displayed in Table 7.

First | SVO | SVO | OVS | OVS
---|---|---|---|---
Second | S die O V | O die S V | S die O V | O die S V
Reading | (_subj-rel_) | (_obj-rel_) | (_obj-rel_) | (_subj-rel_)
_NPN_ |  |  |  |
Irrev. | 89.98 | 91.67 | N/A | N/A
Strong | 65.64 | 74.45 | 40.02 | 30.98
Weak | 61.31 | 64.43 | 46.59 | 45.55
_UD_ |  |  |  |
Irrev. | 89.95 | 71.89 | N/A | N/A
Strong | 72.92 | 24.43 | 19.71 | 65.56
Weak | 67.95 | 17.63 | 14.43 | 64.52
Table 7: Accuracy results for three different relative clauses with context,
i.e. the prior sentence is attended to by the language model prior to parsing,
after finetuning different parts of the parser model on the task itself.

In these results we observe that it is indeed possible to leverage the
training task to even out the parsers’ bias through grounding, leaving the
neural proof net parser accuracy evenly distributed over cases of subject-
relative and object-relative interpretations. On the other hand the UD parser
does not adapt to the task that well and retains the strong bias toward a
subject-relative interpretation. Overall we observe that the price one pays
for the increased accuracy in cases of the object-relative interpretation, is
a signficant drop of performance in the subject-relative case, showing that
developing a balanced parser is no easy task.

##  6 Discussion

In the setup of our experiments, we were careful to develop test cases that
target the parsers in a few different settings. Rather than expecting high
performance overall, the aim of the experiment is to both measure the
prevalence of structural bias in the parser, as well as measuring to what
extent such bias can be mitigated, if present. As such, the experimental
results shouldn’t be taken as proof that the parsers in themselves are
necessarily insufficient.

Rather, it should be taken as a point to argue that parsers should generally
take ambiguity into account, and while lexical ambiguity can be addressed by
means of a neural model iterating over a balanced training dataset, the
inability to accommodate syntactic ambiguity, both in the training corpora
used, as well as in the parser architectures involved, poses a problem that
our experiments confirm.

Aside from the methodological viewpoint above, one may argue that we need not
care about the presented experiment as we could simply finetune underlying
language model together with the parser on top and achieve high performance on
the experiments. While it is true that finetuning the BERT model alongside the
parser leads to peak performance, this is most likely due to the language
model picking up on positional information quickly, and not due to the model
becoming a better parser. To back this claim, we measure parser accuracy
metrics on the original training corpus, for finetuned models that only adapt
the parser or include the underlying language model. The metrics are displayed
in Table 8.

| _NPN_ |  | _UD_
---|---|---|---
| Tag | Frame |  | UAS | LAS
Parser only | 72.54 | 10.32 |  | 87.41 | 85.92
LM+Parser | 73.13 | 9.77 |  | 80.39 | 78.58
Table 8: Parser accuracies on the test set of Lassy Small, after finetuning on
the disambiguation experiment. The top row gives the results for the actual
finetuned models that we evaluate, where the bottom row indicates accuracies
for models where the language model is included in the finetuning process.

For the neural proof net parser, we observe that performance drastically
declines in both cases, with larger decline for the case where the language
model’s parameter were included in the finetuning process. The UD parser on
the other hand does not decrease in performance so much, but has a strong
decline once the language model is included. This highlights two points:
first, given that the neural proof net parser adapted itself better to the
task of our main experiment, we conclude that the price to pay for task
adaptation is a decline in overall parser performance. Second, we argue that
including the language model in finetuning leads to overfitting on the task,
and reduces the parser’s overall accuracy.

##  7 Conclusion

By introducing a synthetic test set of naturalistic Dutch relative clauses, we
carried out an experiment to investigate the sensitivity to structural bias of
two parsing architectures that are both based on a BERT-style language model.
The experiments show that both parser pick up on structural preference for a
subject-relative reading of the relative clause, following a strong
statistical bias coming from the data they were trained on. Further
experimentation shows that a more complex neurosymbolic parsing regime adapts
more easily to a bias correcting finetuning setup than a universal
dependencies parser implemented as a sequence labelling model. However, in
both cases performance on the task is severely below expectation, and we hope
that this work inspires further work on careful data augmentation and parser
development.

## References

  * Aina and Linzen (2021) Laura Aina and Tal Linzen. 2021.  The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation.  In _Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 42–57, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Allein et al. (2020) Liesbeth Allein, Artuur Leeuwenberg, and Marie-Francine Moens. 2020.  Binary and multitask classification model for Dutch anaphora resolution: Die/dat prediction.  _arXiv preprint arXiv:2001.02943_.
  * Arehalli et al. (2022) Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022.  Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities.  In _Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)_ , pages 301–313, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Bouma (2021) Gosse Bouma. 2021.  Probing for dutch relative pronoun choice.  _Computational Linguistics in the Netherlands Journal_ , 11:59–70.
  * Clark (2021) Stephen Clark. 2021.  Something old, something new: Grammar-based CCG parsing with transformer models.  _CoRR_ , abs/2109.10044.
  * Delobelle et al. (2020) Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020.  RobBERT: a Dutch RoBERTa-based Language Model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3255–3265, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Haagen et al. (2022) Tessel Haagen, Loïs Dona, Sarah Bosscha, Beatriz Zamith, Richard Koetschruyter, and Gijs Wijnholds. 2022.  Noun phrase and verb phrase ellipsis in dutch: Identifying subject-verb dependencies with bertje.  _Computational Linguistics in the Netherlands Journal_ , 12:49–63.
  * Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020.  A systematic assessment of syntactic generalization in neural language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1725–1744, Online. Association for Computational Linguistics.
  * Kogkalidis et al. (2020) Konstantinos Kogkalidis, Michael Moortgat, and Richard Moot. 2020.  Neural proof nets.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 26–40, Online. Association for Computational Linguistics.
  * Kogkalidis et al. (2023) Konstantinos Kogkalidis, Michael Moortgat, and Richard Moot. 2023.  SPINDLE: Spinning raw text into lambda terms with graph attention.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , Dubrovnik, Croatia. Association for Computational Linguistics.
  * Kogkalidis and Wijnholds (2022) Konstantinos Kogkalidis and Gijs Wijnholds. 2022.  Discontinuous constituency and BERT: A case study of Dutch.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3776–3785, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Moortgat et al. (2020) Michael Moortgat, Mehrnoosh Sadrzadeh, and Gijs Wijnholds. 2020.  A Frobenius Algebraic Analysis for Parasitic Gaps.  _Journal of Applied Logics_ , 2631(5):823.
  * Moortgat and Wijnholds (2017) Michael Moortgat and Gijs Wijnholds. 2017.  Lexical and derivational meaning in vector-based models of relativisation.  In _Proceedings of the 21st Amsterdam Colloquium_ , page 55. ILLC, University of Amsterdam.
  * van Noord (2010) Gertjan van Noord. 2010.  Self-trained bilexical preferences to improve disambiguation accuracy.  _Trends in Parsing Technology: Dependency Parsing, Domain Adaptation, and Deep Parsing_ , pages 183–200.
  * van Noord et al. (2013) Gertjan van Noord, Gosse Bouma, Frank Van Eynde, Daniël de Kok, Jelmer van der Linde, Ineke Schuurman, Erik Tjong Kim Sang, and Vincent Vandeghinste. 2013.  _Large Scale Syntactic Annotation of Written Dutch: Lassy_, pages 147–164. Springer Berlin Heidelberg, Berlin, Heidelberg.
  * Resnik (1997) Philip Resnik. 1997.  Selectional preference and sense disambiguation.  In _Tagging Text with Lexical Semantics: Why, What, and How?_
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in BERTology: What we know about how BERT works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866.
  * Sinclair et al. (2022) Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. 2022.  Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations.  _Transactions of the Association for Computational Linguistics_ , 10:1031–1050.
  * Strzyz et al. (2019) Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019.  Viable dependency parsing as sequence labeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 717–723, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019\.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _International Conference on Learning Representations_.
  * Vilares et al. (2020) David Vilares, Michalina Strzyz, Anders Søgaard, and Carlos Gómez-Rodríguez. 2020.  Parsing as pretraining.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9114–9121.
  * de Vries et al. (2019) Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019.  BERTje: A Dutch BERT model.  _arXiv preprint arXiv:1912.09582_.
  * Wijnholds (2023) Gijs Wijnholds. 2023.  Assessing monotonicity reasoning in Dutch through natural language inference.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1494–1500, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Wijnholds and Moortgat (2021) Gijs Wijnholds and Michael Moortgat. 2021.  SICK-NL: A dataset for Dutch natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1474–1479, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Hongming Zhang, Hantian Ding, and Yangqiu Song. 2019.  SP-10K: A large-scale evaluation set for selectional preference acquisition.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 722–731, Florence, Italy. Association for Computational Linguistics.