# Improving Explicit Spatial Relationships in Text-to-Image Generation through
an Automatically Derived Dataset

Ander Salaberria HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Gorka Azkune HiTZ Center - Ixa, University of the Basque Country UPV/EHU  Oier
Lopez de Lacalle HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Aitor Soroa HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Eneko Agirre HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Frank Keller University of Edinburgh

###### Abstract

Existing work has observed that current text-to-image systems do not
accurately reflect explicit spatial relations between objects such as _left
of_ or _below_. We hypothesize that this is because explicit spatial relations
rarely appear in the image captions used to train these models. We propose an
automatic method that, given existing images, generates synthetic captions
that contain 14 explicit spatial relations. We introduce the Spatial Relation
for Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs
for training, and more than 60 thousand captions for evaluation. In order to
test generalization we also provide an _unseen_ split, where the set of
objects in the train and test captions are disjoint. SR4G is the first dataset
that can be used to spatially fine-tune text-to-image systems. We show that
fine-tuning two different Stable Diffusion models (denoted as SDSR4G) yields
up to 9 points improvements in the VISOR metric. The improvement holds in the
_unseen_ split, showing that SDSR4G is able to generalize to unseen objects.
SDSR4G improves the state-of-the-art with fewer parameters, and avoids complex
architectures. Our analysis shows that improvement is consistent for all
relations. The dataset and the code are publicly available.111Url:
https://github.com/salanueva/sr4g

Improving Explicit Spatial Relationships in Text-to-Image Generation through
an Automatically Derived Dataset

##  1 Introduction

Text-to-image generators such as Midjourney, Stable Diffusion Rombach et al.
(2022) and Dalle-3 Betker et al. (2023) have recently made rapid advances and
generated a lot of interest. However, those systems are still far from being
perfect and show some important weaknesses. For instance, as observed by
Gokhale et al. (2023) and Cho et al. (2023b) among others, current text-to-
image generators do not represent well explicit spatial relations like _left
of_ or _below_ , which limits their capabilities for important applications
like text-based image editing Kawar et al. (2023).

We hypothesize that the poor performance for explicit spatial relations is due
to the lack of such relations in the datasets used to train those models. To
support our hypothesis we analysed the LAION-2B dataset Schuhmann et al.
(2022), which has been used to train the state-of-the-art open source model
Stable Diffusion. LAION-2B takes the captions from alt-text fields of images
on the web. We automatically searched for explicit spatial relations (_left_ ,
_right_ , _below_ and so on) and found that only $0.72\%$ of cations contain
the target words. Furthermore, $64.1\%$ of these relations are _left_ and
_right_ , which cannot be captured by image generators, as random horizontal
flips are applied to images during training.

Figure 1: Fine-tuning Stable Diffusion on our SR4G dataset improves results
significantly (two versions of SD shown), surpassing the state of the art in
spatial-aware systems (see Section 4).

Motivated by the lack of captions with spatial relations, we focus on the
training data to improve current end-to-end diffusion models; this is
complementary to proposed architectural modifications on the system itself Cho
et al. (2023b); Feng et al. (2023). More concretely, we propose an approach to
automatically generate synthetic captions which contain explicit spatial
relations with paired real images. Leveraging the object annotations in COCO
Lin et al. (2014) and heuristic rules to infer the spatial relation between
two bounding boxes, we build a dataset of real images paired with synthetic
captions, called Spatial Relations for Generation (SR4G).

We use SR4G to fine-tune two Stable Diffusion models, assuming that exposure
to image-caption pairs with explicit spatial relations will enhance the
capabilities of the models to represent those relations. To evaluate our fine-
tuned models and compare to the unmodified base models, we use the recently
proposed VISOR metric Gokhale et al. (2023), which we extend to support more
spatial relations.

The contributions of this paper are the following: (1) We release SR4G, the
first benchmark that allows to fine-tune, develop and evaluate the spatial
understanding capabilities of text-to-image models for 14 explicit relations;
(2) Our experiments show that fine-tuning Stable Diffusion on SR4G improves
the understanding of spatial relations and provides more accurate images; (3)
The improvement holds even when tested on unseen objects, showing that the
models are able to learn the relations, generalizing to unseen objects; (4)
The results exceed the state-of-the-art in spatial understanding for image
generation Cho et al. (2023b); Feng et al. (2023) with fewer parameters and
avoiding complex architectures or Large Language Models.

##  2 Related Work

Many text-to-image systems have been proposed in the last few years. In
general, we can distinguish between those based on auto-regressive transformer
architectures, such as the original Dall-E Ramesh et al. (2021), the multi-
task system OFA Wang et al. (2022) or CogView2 Ding et al. (2022); and those
based on diffusion models, pioneered by GLIDE Nichol et al. (2022), which
evolved into current latent diffusion models such as Stable Diffusion Rombach
et al. (2022) and Attend-and-Excite Chefer et al. (2023).

Although the results of text-to-image systems keep improving, recent work has
shown that their performance for explicit spatial relations is low Gokhale et
al. (2023); Cho et al. (2023b); the models struggle to correctly draw textual
descriptions like a cat on top of a table. To overcome these limitations,
VPGen Cho et al. (2023b) and LayoutGPT Feng et al. (2023) propose pipeline
systems, combining Large Language Models to generate layouts from textual
prompts and layout-to-image generators such as GLIGEN Li et al. (2023). The
difference between both systems is that VPGen fine-tunes Vicuna-13B Chiang et
al. (2023) to generate layouts from textual descriptions, whereas LayoutGPT
relies on Llama-2-7B Touvron et al. (2023) and in-context learning for the
same purpose.222Originally they use LLMs from the OpenAI GPT family, but they
have released a publicly available Llama-2 based variant of LayoutGPT, which
we use in this work.

To avoid the use of complex and large pipeline systems, Yang et al. (2023)
propose ReCo, an end-to-end system which uses layout descriptions in the
input. In this paper, we also focus on end-to-end systems, but we avoid
inserting layout information into the input, as this imposes a substantial
burden on users compared to simple text inputs.

To evaluate the performance of text-to-image generators for explicit spatial
relations, dedicated datasets have been created, since commonly used datasets
like COCO Lin et al. (2014), CC12M Changpinyo et al. (2021) or LAION Schuhmann
et al. (2022), contain very few examples of explicit spatial relations. For
example, Gokhale et al. (2023) propose the $SR_{2D}$ dataset, composed of
synthetic captions created combining two objects in the COCO object vocabulary
and four explicit spatial relations. $SR_{2D}$ only contains captions and it
is thus not amenable for training. Similarly Feng et al. (2023) published the
Numerical and Spatial Reasoning dataset (NSR-1K) which does include caption-
image pairs. The spatial part contains only 1021 image-caption pairs (738 for
train and 283 for test, no development) for 4 relations, insufficient for
accurate evaluation and too small for training.

Our paper proposes a new dataset with synthetic captions and paired images
which can be used to train and evaluate spatial understanding of text-to-image
generation systems, containing 14 different spatial relations and including
9.9 million image/caption pairs (Section 3). Finally, for evaluating the
generated images, we follow Gokhale et al. (2023); Feng et al. (2023); Cho et
al. (2023b) and use an off-the-shelf object detector to extract bounding boxes
and compute the spatial relation between detected objects.

##  3 SR4G: A new synthetic dataset for explicit spatial relation generation

Given the shortcomings of previous datasets, we propose to generate meaningful
synthetic captions for real images, and use them to build the SR4G dataset
(Spatial Relations for Generation). We increase the number of spatial
relations used in previous work Gokhale et al. (2023); Cho et al. (2023b);
Feng et al. (2023) including not only projective or scale relations, but also
topological ones. The full list of unambiguous spatial relations we used is as
follows:

Projective: _left of_ , _right of_ , _above_ and _below_.

Topological: _overlapping_ , _separated_ , _surrounding_ and _inside_.

Scale: _taller_ , _shorter_ , _wider_ , _narrower_ , _larger_ and _smaller_.

Our objective is to build a dataset for training, development and evaluation.
For training, we need image-caption pairs, but for evaluation, captions with
spatial relations are enough, since, following previous work Gokhale et al.
(2023); Cho et al. (2023b), the outputs of the image generator are not
evaluated against real images. The evaluation method is described in Section
3.4.

###  3.1 Captions for evaluation

We first generate a set of spatial triplets of the form $\langle$subject,
relation, object$\rangle$. We build our initial set of triplets using all
pairwise combinations of the 80 objects in the vocabulary of COCO Lin et al.
(2014), yielding $3,160$ object pairs, and combining each pair with all of our
14 spatial relations, resulting in 88,480 spatial triplets.

However, some spatial triplets in the initial set are not natural. For
example, it is very difficult to find natural images for triplets like
$\langle$skis, above, toothbrush$\rangle$ or $\langle$truck, inside,
cat$\rangle$. We want to remove those unnatural triplets from our dataset to
focus on triplets that appear in natural images. Therefore, we identify all
triplets that appear at least once in the training split of the COCO dataset
and use that subset to generate our evaluation captions, which consists of
$68.8\%$ of the entire set of triplets (60,836 triplets).

Using hand-designed templates to be as simple as possible (Appendix A.1), we
generate the final evaluation captions from the set of spatial triplets
(Figure 4 shows some examples). Those captions reflect only the spatial
relations between two objects, avoiding to include any other textual details.

###  3.2 Image-caption pairs for training

For training, we need captions with explicit spatial relations and real images
in which those relations are depicted. We use the COCO 2017 training split to
collect real images with object annotations and define a methodology to
generate first spatial triplets from those images, and then textual captions
derived from those triplets.

Given an image $I$ and a list of $n$ objects
$O_{I}=\\{o_{1},o_{2},\ldots,o_{n}\\}$ belonging to $I$, the goal is to
generate a triplet with a valid spatial relation $r$ between two objects in
$O_{I}$: $o_{s}$ and $o_{o}$, where $s,o\in\\{1,\ldots,n\\}$. For each object
$o_{i}$, we know its respective label $l_{i}$ and bounding box (_bbox_)
$bb_{i}=\\{x_{i}^{0},y_{i}^{0},x_{i}^{1},y_{i}^{1}\\}$, that is, four
coordinates that define the position and size of $o_{i}$ in the image.

Therefore, $t_{j}=\langle l_{s},r,l_{o}\rangle$ is a triplet defined in SR4G
that is represented in $I$. We call this set of valid triplets
$T_{I}=\\{t_{1},\ldots,t_{m}\\}$, where $m$ is the number of valid spatial
relations in the given image $I$. This implies that each relation $r$ has to
be linked to a heuristic rule $f_{r}$ where, given the _bboxes_ of two
objects, it determines whether a given triplet is instantiated or not (see Eq.
1). We follow Johnson et al. (2018) and define $f_{r}$ functions, which
represent unambiguous spatial relations between two object bounding boxes (see
Appendix A.2).

$t_{j}=\langle l_{s},r,l_{o}\rangle\in T_{I}\longleftrightarrow
f_{r}(bb_{s},bb_{o})$ (1)

We apply data augmentation strategies (random crops and horizontal flips) to
the original COCO images in order to obtain an image $I$ and its object list
$O_{I}$. Then, we randomly select two objects as $o_{s}$ and $o_{o}$, compute
the list of valid relations using our predefined $f_{r}$ functions, and
randomly select one of these relations, building the $j$-th valid relation of
$I$ without computing the entire $T_{I}$ set: $t_{j}=(l_{s},r,l_{o})$.
Finally, we verbalize the obtained triplet $t_{j}$ using the same hand-
designed templates as for the evaluation captions (Section 3.1).

###  3.3 Dataset splits

Splits | Images | Unique Captions | I/C Pairs
---|---|---|---
Train | Val | Test
Main | 103.4k | 60.8k | 2.5k | 60.8k | 9.9M
Unseen | 83.6k | 46.9k | 2.5k | 8.0k | 4.8M

Table 1: SR4G dataset’s statistics. _Images_ column refers to the number of
images used during training, _Unique triplets_ column represents the amount of
unique triplets, and _I/C pairs_ refers to the number of unique image/caption
pairs that can be generated.

We build two different splits of SR4G, namely the _main_ and the _unseen_
splits. The _main_ split consists of all the spatial triplets/captions of the
SR4G test set (see Section 3.1). The training instances are generated on-the-
fly without any restrictions on the triplets, which means that the same
triplet can be in train, validation and test splits. For the _unseen_ split,
we randomly divide the COCO dataset’s 80 objects into training, validation and
test sets of $|O_{\mathrm{train}}|=45$, $|O_{\mathrm{val}}|=5$ and
$|O_{\mathrm{test}}|=30$ objects, respectively. More specifically, during
training we just take objects from $O_{\mathrm{train}}$ into account when
randomly selecting _bboxes_ to dynamically build spatial captions. For
validation, as there are few combinations that can be built with
$O_{\mathrm{val}}$, we select triplets that contain one of these 5 objects at
least once and do not contain any object that is set aside for the test split.
For testing purposes we use triplets built by only using objects from
$O_{\mathrm{test}}$. Table 1 shows the relevant numbers of our splits (more
details in Appendix A.3).

###  3.4 Evaluation metrics

To evaluate the performance of text-to-image systems for spatial relations, we
use three evaluation metrics proposed by Gokhale et al. (2023):

#### Object Accuracy:

Given a generated image $I^{\prime}$ and two object labels $l_{a}$ and
$l_{b}$, object accuracy measures whether both objects appear in $I^{\prime}$.
We obtain a list of objects for $I^{\prime}$, i.e.,
$L_{I^{\prime}}=\\{l_{1},\ldots,l_{n}\\}$, by using an off-the-shelf open-
vocabulary object detector, OWL-ViT Minderer et al. (2022). This metric is
useful for analyzing the object generation capabilities of an image generator,
as it does not take the relation $r$ into account.

$\mathrm{OA}(I,l_{a},l_{b})=\begin{cases}1&\mathrm{if}\hskip
4.26773ptl_{a},l_{b}\in O_{I^{\prime}}\\\ 0&\mathrm{else}\end{cases}$ (2)

#### VISOR:

Given a generated image $I^{\prime}$ and a spatial triplet
$t=(l_{a},r,l_{b})$, VISOR measures whether both objects appear and if the
spatial relation $r$ is valid between them. Function $f_{r}$ takes the
bounding boxes of both objects ($bb_{a}$ and $bb_{b})$ and compares them to
check if the triplet is valid. Bounding boxes are provided by the object
detector. VISOR increases both when the model generates the requested objects
and when the ratio of correctly generated relations increases, showing the
ability of the model in visualising spatial triplets.

$\mathrm{VISOR}(I,t)=\begin{cases}1&\parbox[t]{85.35826pt}{$\mathrm{if}\hskip
2.84544ptl_{a},l_{b}\in L_{I^{\prime}}\wedge f_{r}(bb_{a},bb_{b})$}\\\
0&\mathrm{else}\end{cases}$ (3)

#### VISORCond:

This is the proportion of correctly generated spatial triplets, taking into
account only images in which both objects are generated.

Given that our contribution focuses on spatial understanding, we focus on
VISORCond, as it quantifies the ability of the model to represent spatial
relations correctly without considering its object generation capability. It
is the most informative measure, specially when comparing between systems
which might have different object generation abilities, as it isolates the
understanding of spatial relations. We thus use it as our main performance
metric in the experiments, although we also report the other two metrics,
while extending the number of spatial relations from 4 to 14,

##  4 Experiments and Results

In this section we show that end-to-end models improve their capability of
depicting spatial relations when they are fine-tuned with synthetic training
examples. Furthermore, we find that our fine-tuned models SDSR4G generalize to
unseen objects during fine-tuning.

###  4.1 Experimental set-up

Models. We use Stable Diffusion (SD) as the base model, as it shows the best
performance on spatial relation generation among publicly available end-to-end
models Gokhale et al. (2023). We use two different versions of Stable
Diffusion: SD v1.4 and SD v2.1, which generate images of 512x512 and 768x768
pixels, respectively.

Training. To fine-tune SD models on SR4G, we use the original loss function
proposed by Rombach et al. (2022), i.e., the mean square error over latent
noise representations. We fine-tune SD models for 100k training steps with an
effective batch-size of 64 instances, evaluating on the validation split every
5k steps. After training is complete, we select the checkpoint with the
highest VISORCond value on the validation split. Following Gokhale et al.
(2023), we generate four images per spatial relation in all of our evaluations
for consistency. More details can be found in Appendixes B and C.

###  4.2 Main results

Model | VISORCond $\uparrow$ | VISOR $\uparrow$ | OA $\uparrow$
---|---|---|---
_Main split_
SD v1.4 | 60.9 | 17.6 | 29.0
SD v2.1 | 64.0 | 27.4 | 42.8
SDSR4G v1.4 | 69.0 | 26.8 | 38.9
SDSR4G v2.1 | 69.5 | 31.7 | 45.6
_Unseen split_
SD v1.4 | 60.1 | 17.3 | 28.7
SD v2.1 | 64.0 | 28.4 | 44.4
SDSR4G v1.4 | 68.9 | 23.7 | 34.4
SDSR4G v2.1 | 69.4 | 29.4 | 42.4

Table 2: Results obtained for the _main_ and _unseen_ splits of SR4G. Base
models SD v1.4 and v2.1 are shown alongside with fine-tuned SDSR4G models.

Table 2 shows the results for our base and fine-tuned models for both SR4G
splits, with the best results according to the main comparison metric in bold.

Main split:  We observe that the SDSR4G models improve all metrics respect to
the base SD models, increasing both object and spatial relation generation
capabilities considerably. These results are in line with our initial
hypothesis, proving that the exposure to image-caption pairs with explicit
spatial relations improves spatial relation generation. Our results show that
SDSR4G v1.4 and v2.1 have almost the same spatial capabilities, but v2.1
excels for object rendering. Notice that the differences of the base SD models
are much bigger.

Unseen split: To analyse whether the improvements of SDSR4G on the _main_
split come from learning specific correlations between pairs of objects, or
between objects and spatial relations, we check the results on the _unseen_
split. The _unseen_ split uses different objects in train and test, and it is
thus designed to decouple objects from spatial relations, allowing us to focus
on the performance for spatial relations in isolation. In Table 2, we see that
both versions of SDSR4G consistently improve the VISORCond and VISOR metrics
over the base SD systems, also for the _unseen_ split. It is specially
interesting that VISORCond, which is not influenced by object accuracy, is
almost the same as for the _main_ split. That means that our models are
generalizing to unseen objects during the fine-tuning step. The behaviour of
both versions is very similar to the _main_ split.

Image quality: As we are using synthetic captions to train, we make sure that
the image generation capabilities of these models do not worsen over training.
Therefore, we monitor the Fréchet Inception Distance (FID) Heusel et al.
(2017) between the model’s generated images from human annotated captions
(retrieved from the COCO 2017 validation split) and their respective real
images. During all of our experiments FID values have been constant and have
not worsen after training. A random set of examples can be seen in Figure 4.

Model | Par. | VISORCond $\uparrow$ | VISOR $\uparrow$ | OA $\uparrow$
---|---|---|---|---
_Main split_
LayoutGPT | 8.1B | 64.7 | 24.7 | 38.1
VPGen | 14.1B | 67.7 | 34.5 | 51.0
SD v2.1 | 1.3B | 64.0 | 27.4 | 42.8
SDSR4G v2.1 | 1.3B | 69.5 | 31.7 | 45.6
_Unseen split_
LayoutGPT | 8.1B | 64.7 | 24.7 | 38.1
VPGen $\dagger$ | 14.1B | 68.4 | 37.0 | 54.1
SD v2.1 | 1.3B | 64.0 | 28.4 | 44.4
SDSR4G v2.1 | 1.3B | 69.4 | 29.4 | 42.4

Table 3: Comparison to the state of the art, including model size for both
splits. $\dagger$ VPGen is contaminated, as it was trained on layouts
containing spatial triplets that appear in our test split.

###  4.3 Comparison with the state of the art

We also compare against two recent state-of-the-art pipeline models: LayoutGPT
and VPGen. The backbone Large Language Model (LLM) of VPGen has already been
fine-tuned for layout generation,333They use three different datasets to
obtain caption-layout pairs to fine-tune the LLM: Flickr30K entities Plummer
et al. (2015), COCO instances 2014 Lin et al. (2014), and PaintSkills Cho et
al. (2023a). so we use VPGen with no further adaptation. Note that the layout
generation module of VPGen has been trained on COCO, and thus contains the
objects underlaying our test sets. In the case of LayoutGPT, adaptation is
performed with in-context learning. We thus define a set of instances that
will be used as in-context examples to condition the 7B parameter Llama-2 LLM.
For this purpose, we randomly extract 400 caption-layout pairs per different
relation from our SR4G dataset, and build a set of 5.6k instances of caption-
layout pairs. For inference, $k=8$ examples are chosen by computing the CLIP-
based similarity Radford et al. (2021) between the input caption and the set
of in-context examples, retrieving the top-$k$ most similar examples and using
them to condition the model to generate the proper layout.

Table 3 shows the obtained results for both SR4G splits. The same trend is
observed, i.e. SDSR4G v2.1 clearly outperforms both state-of-the-art pipeline
systems in terms of VISORCond, which measures the correctness of the spatial
relation when both objects are generated. The improvement is especially
important considering that both pipeline systems are significantly larger in
terms of parameters, with a more complex architecture involving LLMs, and that
both are specifically designed to generate scene layouts.

The table also shows the two auxiliary metrics, with VPGen obtaining the best
results for object accuracy and VISOR. That is expected, since VPGen has been
trained specifically for object generation, and VISOR is calculated over all
the recognised objects. In fact, the better VISOR results are only due to
better object accuracy, as our method produces better spatial configurations
after factoring out object accuracy from VISOR (VISORCond). Also note the
contamination issue for the _unseen_ split, as the text-to-layout step of
VPGen has been fine-tuned on COCO. This implies that VPGen has seen text-
layout pairs using the entire set of objects, having been trained on all the
objects in our test set.

##  5 Analysis

Type | Relation | Main Split | Unseen Split
---|---|---|---
Projective | _Left of_ | 70.3 (+7.0) | 69.8 (+8.8)
_Right of_ | 72.4 (+8.0) | 67.9 (+3.9)
_Above_ | 72.0 (+4.5) | 70.4 (+2.2)
_Below_ | 71.4 (+4.5) | 70.3 (+2.8)
Topological | _Overlapping_ | 86.9 (-4.9) | 84.0 (-5.2)
_Separated_ | 79.5 (+17.0) | 84.8 (+18.5)
_Surrounding_ | 29.8 (+2.3) | 21.7 (-2.1)
_Inside_ | 43.4 (-7.4) | 39.2 (-6.4)
Scale | _Taller_ | 71.2 (+1.6) | 75.6 (+5.0)
_Shorter_ | 67.5 (+8.5) | 69.0 (+11.9)
_Wider_ | 71.6 (+4.3) | 73.0 (+6.9)
_Narrower_ | 69.3 (+9.3) | 67.1 (+5.0)
_Larger_ | 71.5 (+0.5) | 74.7 (+1.9)
_Smaller_ | 65.2 (+12.7) | 63.3 (+13.5)

Table 4: VISORCond values per relation obtained by SDSR4G v2.1. The difference
in VISORCond between SD v2.1 and fine-tuned SDSR4G is given in brackets.

We show an extensive analysis of the consequences of fine-tuning on SR4G,
covering performance per relation, biases for opposite relations, performance
by frequency of triplets and qualitative examples.

###  5.1 Analysing performance per relation

In Table 4 we show VISORCond values per spatial relation for SDSR4G v2.1 (our
best model), both in the _main_ and _unseen_ splits.

First, we observe that all projective relations significantly improve for both
splits. The improvement is bigger for _left of_ and _right of_. That might be
due to random horizontal flips applied only to the images during the training
of SD models, which are expected to damage the model’s ability to correctly
learn those relations.

Figure 2: The horizontal axis depicts the difference of VISORCond values
between relation pairs with opposing meanings defined on each side of the
vertical axis. Results for SD and SDSR4G v2.1 on the _unseen_ split.

Topological relations show a more variable behaviour. In the case of
_separated_ , our unique topological relation that does not involve generating
overlapping objects, SDSR4G is capable of improving its performance by up to
18.5 points VISORCond. However, for _overlapping_ , fine-tuning is not
helpful. SD v2.1 already knows how to generate images with the _overlapping_
relation, achieving VISORCond values of 91.8 and 89.2 in both test splits. On
the other hand, _surrounding_ and _inside_ seem to be especially hard. The
VISORCond values are low for the SD model and fine-tuning even makes them
worse (especially for _inside_). This is a limitation of our current approach,
and different training strategies must be explored to tackle this issue.

Finally, SDSR4G improves for all scale relations. It is curious to observe
that _taller_ , _wider_ and _larger_ perform better than their opposites, even
though the improvements over the base SD model are more modest. That suggests
that the base SD model might have a bias towards those spatial relations.

(a) Results using _main_ splits.

(b) Results using _unseen_ splits.

Figure 3: Correlation between the frequency of SR4G triplets in COCO training
instances (shown in the logarithmic horizontal axis) and their respective
VISORCond results for SD v2.1 and SDSR4G v2.1. Triplets are grouped by
frequency for visibility.

###  5.2 Analysing biases for opposite relations

Most of our relations have an opposite relation, i.e., _right of_ is the
opposite of _left of_. There are a total of six pairs of opposites in our
relation set, which are listed in Figure 2 along with the difference in
performance for these pairs before and after fine-tuning using the _unseen_
split.

We want to see whether performance biases between opposites are reduced by
fine-tuning. Figure 2 shows strong preferences of our base model SD v2.1 (in
Appendix D, we show that those differences are correlated with the rate of
appearance of each relation in the pretraining dataset of the SD models). We
can also observe that SDSR4G v2.1 significantly reduces the difference in
VISORCond between all relation pairs (except for _wider_ and _narrower_),
showing that fine-tuning reduces the inherent biases of the base model.

###  5.3 Performance by frequency of triplets

As SR4G is derived from natural images, some triplets are more frequent than
others. To measure how the frequency of training triplets affects the results
of our fine-tuned models, in Figure 3, we depict the VISORCond values of SD
v2.1 and SDSR4G v2.1 depending on the frequency of each triplet in the COCO
training set.

Figure 3 shows the results for the _main_ split. In this case, the image
generator has seen test triplets during training and, as expected, the more
frequent these triplets, the greater the improvement after the fine-tuning. We
can also observe that, even though SD models have not seen COCO images before,
its performance is correlated with our computed frequencies.

On the other hand, Figure 3 shows a similar plot when training and evaluating
on the _unseen_ split. We observe similar correlations as in Figure 3 with
both models. However, now we are evaluating on images generated from unseen
triplets composed by objects that have not been seen during fine-tuning.
Therefore, these results show that it is easier to transfer what is learnt to
the most common triplets, even though we have not trained on them.

###  5.4 Qualitative Analysis

In order to visualize and qualitatively evaluate the generated images, we take
SD v2.1 and SDSR4G v2.1 fine-tuned on the _main_ split. We discard the most
common and uncommon spatial triplets. The rationale is that the most common
triplets often contain easy-to-generate relations (e.g., $\langle$truck,
larger, dog$\rangle$) as generating both objects is enough to instantiate the
relation itself, whereas the least frequent ones do not seem natural and would
not be used in a prompt (e.g., $\langle$bus, shorter, traffic light$\rangle$).
Therefore, we randomly pick triplets that occur between 100 and 1,000 times in
COCO annotations (we obtain that range from the frequency analysis in Figure
3). We start generating images using random captions. We keep the first nine
image pairs where both objects are generated correctly. Those nine pairs can
be found in Figure 4, where we also indicate whether the spatial relation in
the caption is depicted correctly or not.

Figure 4: Image generation examples by SD v2.1 and SDSR4G v2.1 fine-tuned on
the _main_ split. Following our relation-specific heuristics, if the relation
in the caption is correctly depicted, we indicate this with a green tick.
Otherwise, there is a red cross in the top-right corner of the image.

Some of the captions of Figure 4 describe _easy_ spatial relations, such as
number 2, 3, 6, 7 and 9, where usually, if the correct objects are generated,
the relation is also correct. SDSR4G generates those relations correctly,
except for 3, which we denoted as a failure because the cup is not fully
visible (the decision is arguable). SD fails for 2, rendering the traffic
light very oddly. Captions 1, 4, 5 and 8 are more demanding: SDSR4G correctly
depicts all the relations (_right of_ twice, _overlapping_ and _separated_),
but SD fails for 1, 5 and 8. The failures are interesting: for 1 and 8, the
spatial relations of the captions might not be the most typical ones in
natural images, and SD struggles. However, for 5 it should be very common to
see dogs and chairs separated, but SD does not follow the caption, which
suggests that the relation _separated_ is not known to SD.

##  6 Conclusions

In this work we define a dataset generation pipeline to build synthetic
captions containing explicit spatial relations from COCO images and
annotations. Fine-tuning diffusion models with these image-caption pairs
outperforms the original diffusion models and also surpasses state-of-the-art
pipeline models for spatial relation generation. We find that SDSR4G
generalizes to unseen objects during fine-tuning. Further analysis shows that
SDSR4G learns to better depict projective and scale relations, reduces the
bias that the original model has for opposite relations, and generalizes
better to spatial triplets that are more frequent in real images.

As future work, we plan to expand our relation set to include depth
information with relations such as _in front of_ and _behind_. We would also
like to explore new ways to collect and annotate natural captions with spatial
relations and evaluate state-of-the-art models with them.

##  7 Limitations

SR4G only contains captions in English, which limits its usage for non-English
languages. To make it multi-lingual, caption generation scripts should be
modified. On the other hand, SR4G is focused on unambiguous spatial relations
defined over bounding box information, since they can be generated and
evaluated automatically using off-the-shelf object detectors and heuristic
rules. In that sense, orientation relations are discarded, even though their
analysis is very interesting. Finally, we focus on 2D spatial relations. To
introduce 3D relations should also be possible, using off-the-shelf depth
estimation systems for images.

## References

  * Betker et al. (2023) James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2(3):8.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568.
  * Chefer et al. (2023) Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. 2023.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_.
  * Cho et al. (2023a) Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023a.  Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3043–3054.
  * Cho et al. (2023b) Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023b.  Visual programming for text-to-image generation and evaluation.  _arXiv preprint arXiv:2305.15328_.
  * Ding et al. (2022) Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. 2022.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  _Advances in Neural Information Processing Systems_ , 35:16890–16902.
  * Feng et al. (2023) Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2023.  Layoutgpt: Compositional visual planning and generation with large language models.  _arXiv preprint arXiv:2305.15393_.
  * Gokhale et al. (2023) Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. 2023.  Benchmarking spatial relationships in text-to-image generation.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30.
  * Johnson et al. (2018) Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018.  Image generation from scene graphs.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1219–1228.
  * Kawar et al. (2023) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Li et al. (2023) Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023.  Gligen: Open-set grounded text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22511–22521.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Minderer et al. (2022) Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. 2022.  Simple open-vocabulary object detection.  In _European Conference on Computer Vision_ , pages 728–755. Springer.
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning_ , pages 16784–16804. PMLR.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8821–8831. PMLR.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2022) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In _International Conference on Machine Learning_ , pages 23318–23340. PMLR.
  * Yang et al. (2023) Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. 2023.  Reco: Region-controlled text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14246–14255.
  * Zhai et al. (2022) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022.  Scaling vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12104–12113.

##  Appendix A Details on SR4G Dataset

In this appendix, we give more details about our _main_ and _unseen_ splits,
as well as defining our hand designed templates and heuristics used to
determine whether an image contains a given spatial relation between two
objects.

###  A.1 Hand designed templates

The templates we use to generate captions from spatial triplets are shown in
Table 5. As can be seen, those templates are designed to be as simple as
possible, omitting attributes and verbs and focusing only on the objects and
their spatial relation. This is very important to analyse spatial
understanding in isolation.

Type | Relation | Template
---|---|---
Projective | _Left of_ |  $\langle$A$\rangle$ to the left of $\langle$B$\rangle$.
_Right of_ |  $\langle$A$\rangle$ to the right of $\langle$B$\rangle$.
_Above_ |  $\langle$A$\rangle$ above $\langle$B$\rangle$.
_Below_ |  $\langle$A$\rangle$ below $\langle$B$\rangle$.
Topological | _Overlapping_ |  $\langle$A$\rangle$ overlapping $\langle$B$\rangle$.
_Separated_ |  $\langle$A$\rangle$ and $\langle$B$\rangle$ separated.
_Surrounding_ |  $\langle$A$\rangle$ surrounding $\langle$B$\rangle$.
_Inside_ |  $\langle$A$\rangle$ inside of $\langle$B$\rangle$.
Scale | _Taller_ |  $\langle$A$\rangle$ taller than $\langle$B$\rangle$.
_Shorter_ |  $\langle$A$\rangle$ shorter than $\langle$B$\rangle$.
_Wider_ |  $\langle$A$\rangle$ wider than $\langle$B$\rangle$.
_Narrower_ |  $\langle$A$\rangle$ narrower than $\langle$B$\rangle$.
_Larger_ |  $\langle$A$\rangle$ larger than $\langle$B$\rangle$.
_Smaller_ |  $\langle$A$\rangle$ smaller than $\langle$B$\rangle$.
Table 5: Templates used to generate synthetic captions.

###  A.2 Heuristic rules

We use heuristic rules to both build the dataset and evaluate the generated
images. Assuming the spatial triplet $\langle l_{s},r,l_{o}\rangle$ and the
bounding boxes of its objects $bb_{s}$ and $bb_{o}$ that appear in an image,
we define the heuristic rule $f_{r}$ of relation $r$ to determine whether the
triplet is fulfilled in the image or not. We set
$bb_{i}=\\{x^{0}_{i},y^{0}_{i},x^{1}_{i},y^{1}_{i}\\}$ by defining the top-
left $\\{x^{0}_{i},y^{0}_{i}\\}$ and bottom-right coordinates
$\\{x^{1}_{i},y^{1}_{i}\\}$ of the bounding-box (_bbox_).

For _left of_ , _right of_ , _above_ and _below_ , we follow the heuristic
rules defined in Gokhale et al. (2023), by computing the centroid of each
_bbox_ $c_{i}=\\{x^{c}_{i},y^{c}_{i}\\}$ and comparing their corresponding
coordinates.

As we expand to 10 more relations, we follow the rules described in Johnson et
al. (2018). In our scale relations we compare either the height (_taller_ and
_shorter_), width (_wider_ and _narrower_) or area (_larger_ , _smaller_)
difference between both _bboxes_. In the cases of _surrounding_ and _inside_ ,
we check whether $bb_{o}$ is contained in $bb_{s}$ or vice versa. Finally,
using the Intersection over Union (IoU) of both _bboxes_ , we say that both
objects are _separated_ if their IoU is 0, and _overlapping_ if their IoU is
positive.

###  A.3 Main and Unseen Splits

$O_{\mathrm{Train}}$
---
_person_ , _car_ , _motorcycle_ , _airplane_ , _train_ , _boat_ , _fire
hydrant_ , _bench_ , _bird_ , _elephant_ , _bear_ , _giraffe_ , _handbag_ ,
_tie_ , _snowboard_ , _baseball bat_ , _baseball glove_ , _surfboard_ , _cup_
, _knife_ , _spoon_ , _apple_ , _sandwich_ , _orange_ , _broccoli_ , _carrot_
, _pizza_ , _donut_ , _chair_ , _couch_ , _potted plant_ , _bed_ , _dining
table_ , _toilet_ , _laptop_ , _mouse_ , _remote_ , _keyboard_ , _oven_ ,
_sink_ , _book_ , _clock_ , _teddy bear_ , _hair drier_ , _toothbrush_
$O_{\mathrm{Val}}$
_umbrella_ , _cake_ , _tv_ , _refrigerator_ , _vase_
$O_{\mathrm{Test}}$
_bicycle_ , _bus_ , _truck_ , _traffic light_ , _stop sign_ , _parking meter_
, _cat_ , _dog_ , _horse_ , _sheep_ , _cow_ , _zebra_ , _backpack_ ,
_suitcase_ , _frisbee_ , _skis_ , _sports ball_ , _kite_ , _skateboard_ ,
_tennis racket_ , _bottle_ , _wine glass_ , _fork_ , _bowl_ , _banana_ , _hot
dog_ , _cell phone_ , _microwave_ , _toaster_ , _scissors_
Table 6: Objects used in train, val and test sets of our _Unseen split_.

Table 6 shows the sets of objects used for training, validation and test in
the _unseen_ split, which we refer to as $O_{\mathrm{train}}$,
$O_{\mathrm{val}}$ and $O_{\mathrm{test}}$, respectively.

There are few combinations that can be built with $O_{\mathrm{val}}$ for
validation in the _unseen_ split, so we select triplets that contain one
object from $O_{val}$ at least once and do not contain any object that is set
aside for the test split. In other words, there are up to
$(2\cdot|O_{\mathrm{train}}|\cdot|O_{\mathrm{val}}|+{|O_{\mathrm{val}}|\choose
2})\cdot 14=6,580$ triplets that fulfil this rule (around 5,326 that naturally
occur in the COCO dataset).

Validation is computationally costly in both splits, as several images have to
be generated to compute the evaluation metrics defined in Section 3.4.
Preliminary experiments showed that generating just 10k images is enough to
get consistent results. Thus, we randomly selected 2.5k spatial captions for
the validation splits for both _main_ and _unseen_ splits (as we generate 4
images per caption).

##  Appendix B Training settings

Hyperparameter | Value
---|---
Training steps | 100k
Batch size | 64
Learning Rate | $10^{-5}$
Optimizer | AdamW
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.999
Adam $\epsilon$ | $10^{-8}$
Weight decay | 0.01
Mixed-precision | bf16
Table 7: Fine-tuning hyperparameters of the diffusion models.

Hyperparameters: In Table 7 we define the hyperparameters used for training.
Learning rate and optimizer parameters are the ones used during the
pretraining of SD models, the other listed hyperparameters have been adapted
to our available infrastructure. We also take advantage of Exponential Moving
Average Kingma and Ba (2015) to update the parameters of the models with an
AdamW optimizer Loshchilov and Hutter (2019) and we do not use any learning-
rate scheduler. We do validation runs every 5k steps and do not set any early-
stopping mechanism.

GPU usage: Due to different memory needs, we use 2 and 4 NVIDIA A100 GPUs to
fine-tune SD v1.4 and SD v2.1 models, respectively. In both cases we use an
effective batch size of 64 by changing the amount of instances assigned to
each GPU. Each of our fine-tunings need 3 days to be completed.

Data augmentation: During training we apply random horizontal flips and random
crops to our images as a data augmentation strategy (resulting in $I^{*}$ and
$O^{j}$). Note that, random horizontal flips are common during the training of
text-to-image models. This implies that spatial relations, such as _left of_
and _right of_ , can not be learnt correctly (as captions are not transformed
according to those flips). Nevertheless, in our case we apply the same
transformations to _bboxes_ , which are used to generate captions
synthetically, keeping this data augmentation strategy while maintaining the
generated caption’s spatial correctness.

Random crops might reduce the number of objects in $O_{I^{*}}$. If there are
less than two objects after a given crop, we redo it up to $max\\_iter$ times
until there are at least two objects in the image.

We also define the hyperparameter $k$ as the number of captions that can be
concatenated to build the image-caption pairs built during training. Table 8
shows the results obtained by concatenating $k\in\\{1,\ldots,5\\}$ captions.
We observe that $k=2$ obtains the best results, and we use this value of $k$
during our entire work.

Nº Captions | VISORCond $\uparrow$ | VISOR $\uparrow$ | OA $\uparrow$
---|---|---|---
1 | 68.1 | 26.5 | 38.9
2 | 69.4 | 27.4 | 39.5
3 | 67.7 | 27.1 | 40.0
4 | 63.7 | 21.9 | 34.3
5 | 63.0 | 22.9 | 36.3

Table 8: We fine-tune SD v1.4 in the _main_ split concatenating different
amounts of captions in the input. These results correspond to the validation
set of our _main_ split.

##  Appendix C Evaluation settings

The evaluation metrics used in this paper use an object detector to determine
whether objects are generated correctly and where are located in the image.
Following Gokhale et al. (2023), we use OWL-ViT, an open-vocabulary object
detector that uses a CLIP Radford et al. (2021) backbone with a ViT-B/32
transformer architecture Zhai et al. (2022). We also set 0.1 as the confidence
threshold of OWL-ViT, which determines how sure the model must be for a given
region of the image to contain a specific object.

As an open-vocabulary object detector, OWL-ViT takes as input the objects we
want to detect and, in order to do so, we use their recommended template ("a
photo of a $\langle$OBJ$\rangle$.") instead of the object label alone.

Due to the variability of images generated by Stable Diffusion, we generate 4
images per evaluation caption. Therefore, we generate 10k images per
validation and a total of 243.3k and 32.1k images to test each model in the
_main_ and _unseen_ splits, respectively.

##  Appendix D LAION Dataset and Spatial Relations

Figure 2 shows that Stable Diffusion models have a strong bias towards some
spatial relations, preferring _taller_ to _shorter_ , for instance. To
complete those results, we also show the same graphic but in the _main_ split,
which exhibits a very similar behaviour (Figure 5). To understand the origin
of those biases, we checked the frequency of each spatial relation in the
LAION-2B dataset (English subset), used to train SD models. Table 9 shows the
appearances of 12 relations, divided in 6 relation pairs with opposite
meanings. Every relation has its number of appearances in LAION into brackets.
For each opposite relation pair, the first column contains the relation that
best works with SD. The third column shows the ratio of appearance between the
preferred relation and its opposite (>1 indicates that the preferred relation
appears more times in LAION than its opposite relation). The results indicate
that there is a clear correlation between the ratio of appearance of a
relation and the bias of SD models. The only exception is the _right_ and
_left_ pair, but both appear similar times and the bias towards _right_ is
very small.

Figure 5: The horizontal axis depicts the difference of VISORCond values
between relation pairs with opposing meanings defined on each side of the
vertical axis. These results correspond to SD and SpaD v2.1 trained and
evaluated using _main_ splits.

Preferred Rel. | Opposite Rel. | Ratio of Appearance
---|---|---
Right (5M) | Left (5.6M) | 0.91
Above (1.6M) | Below (0.7M) | 2.47
Inside (2M) | Surrounding (0.3M) | 7.61
Taller (49.3K) | Shorter (29.4K) | 1.86
Wider (54.6K) | Narrower (5.7K) | 9.62
Larger (0.8M) | Smaller (0.2M) | 3.17

Table 9: Ratio in which the first relation appears more than the other. The
relation in the first column is the preferred one by SD.


# A hierarchical modelling approach to assess multi pollutant effects in time-
series studies

Marta Blangiardo  MRC-PHE Centre for Environment and Health, Department of
Epidemiology and Biostatistics, Imperial College, St Mary’s Campus, Norfolk
Place, London W2 1PG  Corresponding author  Monica Pirani  MRC-PHE Centre for
Environment and Health, Department of Epidemiology and Biostatistics, Imperial
College, St Mary’s Campus, Norfolk Place, London W2 1PG  Lauren Kanapka
Department of Mathematics, Imperial College, South Kensington Campus,
Exhibition Road, London SW7 2AZ  Anna Hansell  MRC-PHE Centre for Environment
and Health, Department of Epidemiology and Biostatistics, Imperial College, St
Mary’s Campus, Norfolk Place, London W2 1PG  Gary Fuller  MRC-PHE Centre for
Environment and Health, Environmental Research Group, King’s College, 150
Stamford Street, London SE1 9NH

###### Abstract

When assessing the short term effect of air pollution on health outcomes, it
is common practice to consider one pollutant at a time, due to their high
correlation. Multi pollutant methods have been recently proposed, mainly
consisting of collapsing the different pollutants into air quality indexes or
clustering the pollutants and then evaluating the effect of each cluster on
the health outcome. A major drawback of such approaches is that it is not
possible to evaluate the health impact of each pollutant. In this paper we
propose the use of the Bayesian hierarchical framework to deal with multi
pollutant concentration in a two-component model: a pollutant model is
specified to estimate the ‘true’ concentration values for if your each
pollutant and then such concentration is linked to the health outcomes in a
time series perspective. Through a simulation study we evaluate the model
performance and we apply the modelling framework to investigate the effect of
six pollutants on cardiovascular mortality in Greater London in 2011-2012.

##  1 Introduction

Short-term air pollution studies aim at evaluating the association between the
day-to-day variation in ambient air pollution and the day-to-day variation in
a health outcome, such as mortality or hospital admissions. Typically this
involves a time-series approach using data from a particular geographical area
that contains daily counts of mortality or morbidity, pollution and
meteorological measurements. To provide a few recent examples, Xie et al. 1
used a data set from Beijing containing counts of daily hospital admissions
and mortality from ischaemic heart disease (IHD), measurements of fine
particulate matter air pollution and potential confounding meteorological
variables such as temperature or relative humidity. Using a Poisson model,
they reported a significant association between the pollutant and IHD. Other
studies using a time-series approach 2, 3, 4 had mixed results in detecting a
short-term association between various air pollutants and a mortality outcome.

It is obvious that the air that we breath contains a number of different
pollutants; however, due to the high correlation between these, the typical
approach used in this field evaluates the health effect of one pollutant (or
at most two pollutants) at a time. Despite this, recently there have been some
attempts to move towards a multi pollutant approach that more realistically
depicts the complexity of the exposure. For instance, Pirani et al. 5 proposed
a Dirichlet process (DP) mixture model to cluster the days by their exposure
profiles. The model jointly estimates the covariate patterns and the health
effect of each cluster. In addition the use of the DP allows the number of
clusters to be determined by the model and the data, guaranteeing extreme
flexibility.

In a similar perspective Bobb et al. 6 proposed a Bayesian Kernel Machine
Regression (BKMR). The idea is to include the pollutants in the model using a
smooth function h that is represented through a kernel function. The authors
focused on Gaussian kernel as it outperformed linear and ridge regression
kernels in simulation studies where h has a complex functional form.

Similarly to the DP approach, the focus of BKMR is to correctly identify the
exposure-response relationship rather than to identify the effect of each
individual pollutant on the outcome. To partially address this, the authors
extended their model to include a framework for variable selection, allowing
the inclusion only of pollutants which have an impact on the response.
However, such approach does not quantify this impact to determine exactly how
much the pollutants are affecting the response.

An alternative approach to deal with multiple pollutants consists of building
a composite air quality index, that summarises their concentration; this is
commonly carried out by most governments and used to inform people of the
health risk posed by air quality. For example, in the UK the Daily Air Quality
Index (DAQI) is based on the highest pollutant concentration out of the
following regulated ones: sulphur dioxide, ozone, particulate matters,
nitrogen dioxide. The concentration is then transformed into pre-determined
bands (low, moderate, high, very high). Recently DAQI has been used to
evaluate the effect of episodes of high air pollution concentration on
respiratory conditions 7.

Finazzi et al. 8 proposed a more sophisticated approach by using a
hierarchical model based on latent variables, that they called dynamic co-
regionalization model. This model aggregates the pollutant data over space and
addresses problems such as missing data and an unbalanced network, where not
all pollutants are measured at every site. The output of this model can then
be used to calculate an index, for instance by taking the maximum, such as the
DAQI does. The authors applied their model to Scottish air pollution data and
used the maximum to calculate a state-wide index over time.

Very recently Huang et al. 9 proposed a two-stage spatio-temporal approach to
evaluate the effects of two pollutants on respiratory hospital admissions in
Scotland. In the first step they estimate annual concentrations from
monitoring stations and output from numerical models and then feed forward the
estimates and their uncertainty to the second step to assess the health
effects. To include the two pollutants in the second stage, avoiding
collinearity, they consider the first pollutant and the residual of the second
after accounting for the first through a linear regression, making the
approach difficult to be extended to more than two pollutants.

Our paper is set in a similar perspective as we develop a two-components
Bayesian hierarchical model that quantifies the health effect of multi
pollutants. In the first component we account for measurement error in the
observed air pollution measurements and for correlation among pollutants;
based on this we estimate the corresponding latent ‘true’ concentration
values. However, our paper novelty lays on its fully Bayesian framework, as
the two components are jointly estimated so that uncertainty from the
concentration estimates can feed forward into the health effect estimates; at
the same time information from the outcome can feedback to the air pollution
estimates. We do rely on the joint estimation process, on the hierarchical
nature of the model and on informative priors on the health effect parameters
to overcome the collinearity among the pollutants, making the framework
extendible to any number of pollutants, hence able to disentangle synergic or
antagonistic effects of pollutants which would be not detectable in the common
single-pollutant modelling framework. The developed approach is used to
evaluate the effect of five pollutants (carbon monoxide - CO, nitrogen dioxide
- NO2, ozone - O3, sulphur dioxide - SO2 and fine particulate matter, smaller
in size than 2.5 $mg/m^{3}$ \- PM2.5) and particle number concentration - PCNT
on daily cardiovascular mortality in Greater London for 2011-2012.

The remainder of the paper is structured as follows: section 2 presents the
data and the model, section 3 introduces the simulation study, while in
section 4 we present the results of our analysis and section 5 covers areas of
discussion and concluding remarks.

##  2 Material and Methods

###  2.1 Data Description

Daily measurements of CO, NO2, O3, SO2, PM2.5 and PCNT were obtained from a
monitoring site in North Kensington, London (UK) over the period 1 January
2011 to 31 December 2012. The London North Kensington site (lat 5131015.78000
N, long 012048.57100 W) is part of both the London Air Quality Network and the
national Automatic Urban and Rural Network and is owned and part-funded by the
Royal Borough of Kensington and Chelsea. The facility is located within a
self-contained cabin on a school ground in a mainly residential area. It has
been used in previous time-series studies on air pollution health effects
10,5,2. The same monitoring site has also been used extensively as a
background measurement site for source apportionment 11 and also to track the
outcome of policies to improve London air pollution 12.

CO, NO2, O3 and SO2 were measured using CEN mandated methods eg EN 14211 for
NO2. Fortnightly calibrations enabled the traceability of measurements to
national meteorological standards. PM2.5 were measured by TEOM-FDMS (Tapered
Element Oscillating Microbalance - Filter Dynamics Measurement System) which
is considered equivalent to the EU reference method. Particle number
concentration was measured by condensation particle counter (TSI 3022).

We focus on these five pollutants as they are already regulated in ambient
air; as a result, they are well monitored, have documented associations to
health outcomes 13 and have been showed to need National Ambient Air Quality
Standards 14. In addition several papers have focused on one or more of these:
for instance Mills et al. 15 presented a systematic review of the effects on
NO2 where particulate matter is also controlled for. Besides the five
pollutants, we also investigate the effect of PCNT, as this metric was
previously associated to adverse short-term health outcome in London 10.

As a health outcome we consider the daily count of mortality due to
cardiovascular diseases (CVD) over the same period obtained from the UK Office
of National Statistics and available through the Small Area Health Statistics
Unit (SAHSU). These cardiovascular causes were derived from the International
Statistical Classification of Diseases, 10th Revision (ICD-10, Chapter I).

To adjust for potential confounding effect of weather variables, we use daily
average temperature and relative humidity obtained from a meteorological
station close to the North Kensington monitoring site. Table 1 provides
descriptive statistics of the variables considered in the analysis.

Table 1: Descriptive statistics of the variables included in the study | Number | Percentiles |
---|---|---|---
| of Days | 10th | 25th | 50th | 75th | 90th | IQR
Mortality | 731 | 28 | 32 | 37 | 42 | 47 | 10
_Meteorological data_ : |  |  |  |  |  |  |
Temperature (${}^{\circ}C$) | 731 | 5.1 | 8.0 | 11.7 | 15.5 | 18.1 | 7.4
Relative Humidity (%) | 731 | 61.6 | 69.6 | 78.0 | 84.2 | 88.5 | 14.5
_Pollutants_ : |  |  |  |  |  |  |
CO ($mg/m^{3}$) | 715 | 0.1 | 0.2 | 0.2 | 0.3 | 0.4 | 0.1
NO2 ($\mu g/m^{3}$) | 706 | 18.2 | 23.2 | 33.3 | 46.9 | 57.9 | 23.6
O3 ($\mu g/m^{3}$) | 695 | 11.4 | 24.3 | 39.1 | 51.1 | 64.9 | 26.8
SO2 ($\mu g/m^{3}$) | 717 | 0.0 | 0.4 | 1.8 | 2.6 | 3.6 | 2.2
PM2.5 ($\mu g/m^{3}$) | 730 | 5.0 | 6.0 | 9.0 | 14.0 | 25.0 | 8.0
PCNT ($p/mm^{3}$) | 636 | 7.8 | 9.7 | 12.1 | 14.9 | 17.9 | 5.2

###  2.2 Model Specification

Our modelling framework consists of two components jointly estimated: a
pollutant model and a health model, which we describe in details in this
section.

####  2.2.1 Pollutant model

We start specifying $Y_{pt}$ as the measured concentration level of pollutant
$p$ ($p=1,...,P=6$) on day $t$ ($t=1,...,T=731$) from the monitoring site. As
different pollutants are typically characterised by different scales we
recommend standardisation to make them comparable. As $Y_{pt}$ is a continuous
variable it is reasonable to assume the following Normal distribution:

$Y_{pt}\sim N(\mu_{pt},\sigma^{2}_{p})$ (1)

where $\sigma^{2}_{p}$ is the measurement error variance, which is specific
for each pollutant. On $\mu_{pt}$ a linear model is specified as follows:

$\mu_{pt}=\gamma_{0p}+\gamma_{1p}\text{X}_{\text{temp,$t$}}+\gamma_{2p}\text{X}_{\text{temp,$t$}}^{2}+\gamma_{3p}\text{X}_{\text{rhum,$t$}}+\gamma_{4p}\text{X}_{\text{rhum,$t$}}^{2}+\theta_{pt}$
(2)

where $\gamma_{0p}$ is the pollutant specific intercept, while
$\bm{\gamma}_{p}$ are the regression coefficients linking the time-dependent
covariates $\bm{X}_{t}$ to the pollutant levels; as descriptive plots suggest
the presence of a non-linear relationship between the covariates and the
pollutant concentration levels (see Figure 1 in Supplementary material), we
include a linear and quadratic effect of both temperature and relative
humidity. In (2) $\\{\theta_{1t},\ldots,\theta_{Pt}\\}$ account for the
residual temporal effects and for the correlation among pollutants; they are
modelled following a multivariate Normal specification with an autoregressive
structure, following Shaddick et al. 16:

$(\theta_{1t},...,\theta_{Pt})^{{}^{\prime}}\sim\text{MVN}\left((\theta_{1,t-\ell},...,\theta_{P,t-\ell})^{{}^{\prime}},\Sigma_{P}\right)$
(3)

where $t-\ell$ provides the temporal lag of $\ell$ days for the $t$-th day.
For each pollutant, the concentration at time $t$ depends on the values at
time $t-\ell$, while the diagonal of the covariance matrix of the errors
$\Sigma_{P}$ allows each pollutant to have a different amount of temporal
dependence, with larger values indicating a smaller dependence. The off-
diagonals represent the temporal dependence between the pollutants, allowing
the model to incorporate and maintain the correlation structure in the
estimation of the ‘true’ pollutant levels.

Note that this specification has the added benefit of providing a natural way
to deal with missing data in the pollutant concentration. As seen in Table 1,
there are some days where the concentration is not available for one or more
pollutants; the model could impute directly the concentration based on the
correlation with the other pollutants and on the temporal dependency.

####  2.2.2 Health Model

The second component of the model links the ‘true’ latent value of the
pollutant concentrations $\mu_{pt}$ with the counts of the health outcome
within a time-series epidemiological framework. Let Ot be the observed number
of CVD deaths for the day $t$, we specify a Poisson distribution as:

$\text{O}_{t}\sim\text{Poisson}(\lambda_{t}\text{E}_{t})$ (4)

where Et represents the expected number of CVD deaths. Following Pirani et al.
5, we take it to be the average mortality over the whole period, hence Et=E.
Then $\lambda_{t}$ represents the relative risk of CVD death on day $t$
compared to the average. In a previous analysis performed on the same data
set, Atkinson et al. 2 considered the association between CVD and 1-day lagged
pollutant concentrations, thus we coherently adopt the same exposure window
setting $\ell=1$. We therefore specify a regression model on the log link
transformed $\lambda_{t}$:

$\log(\lambda_{t})=\beta_{0}+\sum_{p}\beta_{p}\mu_{p(t-1)}+\sum_{i}s(Z_{ti},\psi_{i})+\delta_{I_{t}}+\epsilon_{t}$
(5)

so that $\exp(\beta_{p})$ is the multiplicative change in relative risk of CVD
death for a unit increase in the pollution concentration obtained from (2). To
be able to interpret the health effects on the correct scale, we back
transform the pollutant concentration $\mu_{pt}$ estimated from (2) to the
original scale. In (5) $s(\cdot,\psi_{i})$ denote smooth functions of daily
average temperature and relative humidity, as well as of calendar time to
account for any residual seasonality and long-term trends. These confounding
factors are included in the model through flexible nonparametric penalised
spline functions 17. In particular, we consider a mixed model framework and
following Crainiceanu et al. 18 we specify a low-rank thin plate spline basis
over other options, which tends to show a smaller posterior correlation
between parameters. By letting $Z_{ti}$ be the $i$-th confounder on day $t$,
we have the following spline representation:

$s(Z_{ti},\psi_{i})=\alpha_{i}Z_{ti}+\sum_{k=1}^{K_{i}}b_{ki}|Z_{ti}-\kappa_{ki}|^{3}$
(6)

where $\psi_{i}=(\alpha_{i},b_{1i},...,b_{Ki})^{{}^{\prime}}$ are the
regression coefficients, $Z_{ti}-\kappa_{ki}$ are the set of basis functions
of the cubic spline and $K_{i}$ is the number of knots for confounder $i$,
with knot locations $\kappa_{1i}<\kappa_{2i}<...<\kappa_{Ki}$. Based on
Atkinson et al. 2 we select 3 knots for temperature and relative humidity and
6 for time. Additionally, to account for any holiday effect, we include in the
model the linear term $I_{t}$ which classifies the days according to workday
or weekend/holiday. Finally to account for overdispersion, that is typically
present when a Poisson distribution is assumed on the data, we include an
additional random effect $\epsilon_{t}\sim N(0,\sigma_{\epsilon}^{2})$.

###  2.3 Prior Specification

The last step in the model specification consists of the choice of prior
distributions. Minimally informative Normal distributions are specified on all
the regression coefficients $\gamma_{0}$, $\bm{\gamma}_{p}$, $\beta_{0}$,
$\delta$ and $\bm{\alpha}$, centered on 0 and with a variance equal to 103.
Given the high correlation present among the pollutants, we take advantage of
an informative prior on $\beta_{p}$; we choose a $N(0,0.1)$ covering a range
of values on the relative rate scale from 0.82 to 1.22, which is plausible
with what has been seen in the literature on cardio-respiratory diseases.

On the standard deviation for the measurement error $\sigma_{p}$ and for the
random effect $\sigma_{\epsilon}$ a Uniform prior is specified ranging between
0 and 100, to ensure minimal information.

The covariance matrix $\Sigma_{P}$ is given a $P$-dimensional inverse Wishart
prior, $IW(D,d)$, where $D$ is a symmetric and positive-definite scale matrix
and $d$ is the degrees of freedom parameter. We follow the specification
presented in Lunn et al. 19 and to ensure the weakest information we fix
$d=P$; as the prior mean for the inverse Wishart is $d^{-1}D$, $D$ is chosen
to be $d$ times the prior estimate of the correlation matrix.

We penalise the random coefficients associated to the basis functions,
$b_{ki}$, shrinking them towards zero to avoid over-fitting. We assume a
Normal prior distribution for those coefficients, with mean 0 and unknown
precision specific for each confounder $\sigma^{-2}_{b_{i}}$. This latter
parameter controls the amount of smoothness and is supplemented with a
$\text{Gamma}(a,b)$ prior distribution, where $a=1;b=0.001$.

###  2.4 Implementation and Sensitivity Analysis

The model is run using a MCMC simulative framework in R; we discarded the
first 50,000 iterations of the MCMC and retained the following 10,000 to
estimate the posterior distribution of the parameters. We considered two
chains and checked for convergence of the parameters visually (see Figures 2-4
in Supplementary Material) and analytically (evaluating the MC error below 5%
of the standard deviation of the posterior estimates as well as the Gelman-
Rubin diagnostic tool).

It is important to stress that this is the first paper to consider jointly the
pollutant and the health components; this results in uncertainty on $\mu_{pt}$
affecting the estimates of the relative risks $\beta_{1},\ldots,\beta_{P}$,
while at the same time the information from the outcome is fed backwards into
the latent concentration values. This is a crucial point as in this way the
correlation between the pollutants is naturally accounted for through the
hierarchical structure and through the input from the outcome.

To evaluate the robustness of our modelling framework we changed the prior
specification of all the parameters where a no informative prior was assumed.
In particular on the regression coefficients we specified a Normal
distribution centered on zero and with a variance equal to 106, while the
measurement error and random effect variance were set to Inverse Gammas with
parameters 1 and 0.001; finally we put a Gamma with parameters $a=b=0.001$ on
the precisions of the random coefficients associated to the basis functions.

A key aspect in air pollution time-series health studies is represented by the
inclusion of the smoothing functions for the time-varying confounding factors.
Here, we need to adequately control for their potential non-linear confounding
effect while retaining sufficient information for estimating the exposure
effects. To perform model checking on the knots we also ran the model with 14
knots on time (7 for each year, 20), and 9 on temperature and humidity, which
we think is large enough to account for a high degree of non linearity, while
at the same time not leading to oversmoothing. As model selection tool we used
the Deviance Information Criterion (DIC, Spiegelhalter et al. 21), one of the
suggested methods22 to choose the degree of smoothness for time-series studies
of air pollution and mortality.

##  3 Simulation Study

We carried out a simulation study to evaluate if the proposed modelling
framework is able to estimate the relative risk of highly correlated
pollutants on a health outcomes.

###  3.1 Simulation set-up

We simulated mortality and air pollution concentration for 2000 days and
considered 6 pollutants. For the sake of simplicity we did not include any
confounder factor (e.g. meteorology) in the pollutant or health components of
the model. We fixed the correlation among the pollutants to be equivalent to
that observed on the time-series data from Greater London:

$\hbox{\multirowsetup
P${}_{P}=$}\left[{\begin{array}[]{cccccc}1&0.737&-0.535&0.442&0.515&0.630\\\
&1&-0.606&0.510&0.730&0.659\\\ &&1&-0.260&-0.394&-0.396\\\ &&&1&0.390&0.490\\\
&&&&1&0.420\\\ &&&&&1\\\ \end{array}}\right].$ (7)

The following steps were used to simulate the data on concentration and
outcome:

  1. 1.

Using the above correlation matrix we generated the true pollutant levels
assuming an autoregressive structure of order 1, as specified in (3),
$\mu_{pt}\sim N(\mu_{p(t-1)},\text{P}_{P})$. This represents the gold standard
exposure.

  2. 2.

At the same time we also simulated the measured concentration for the six
pollutants, which we assumed centered on the true latent exposure, but with a
measurement error variance equal to 0.1 ($Y_{pt}\sim N(\mu_{pt},0.1)$).

  3. 3.

We then simulated the daily number of events for a health outcome using a
Poisson distribution, where the mean $\lambda_{t}$ is specified as

$\displaystyle\log(\lambda_{t})=1+0.2\mu_{t1}+0.2\mu_{t2}-0.2\mu_{t3}+0\mu_{t4}+0\mu_{t5}+0\mu_{t6}$

so that we are able to assess if the model can capture true effects as well as
the lack thereof.

  4. 4.

We repeated the process 100 times.

We ran our modelling framework, hereafter called “hierarchical two-component
model” (H2Mjoint), and compared it with a standard Poisson model, here named
as “measurement error model” (ME), where the true concentration is replaced by
the measured one:

$\displaystyle O_{t}$ $\displaystyle\sim$
$\displaystyle\text{Poisson}(\lambda_{t}E_{t})$
$\displaystyle\log(\lambda_{t})$ $\displaystyle=$
$\displaystyle\beta_{0}+\sum_{p}\beta_{p}Y_{p(t-1)}+\sum_{i}s(Z_{ti},\psi_{i})+\delta_{I_{t}}+\epsilon_{t}$
(8)

We used this as benchmark, given that it is the model commonly specified in
epidemiological studies to study short-term health effects of air pollution.
The graphical representation of ME is presented in Figure 1(b) and shows a
direct link between $Y_{pt}$ and $\lambda_{t}$.

The H2Mjoint model jointly estimates the pollutant concentrations and their
health effects through a fully specified Bayesian framework. As an additional
comparison we specified an alternative model where the two components are
fitted separately, called H2M: for this model first (1)-(3) are run and the
posterior distribution for the pollution concentration is estimated. The
distribution is then fed forward into the health component ((4)-(6)), so that
the air pollution effects on health account for the uncertainty which derives
from their estimated concentration. At the same time in H2M, the feedback from
the health outcome is not allowed to influence the air pollutant concentration
estimates. H2Mjoint and H2M are both represented in Figure 1(a): the former
has two links between $\mu_{pt}$ and $\lambda_{t}$, going each in one
direction (uncertainty feeding forward and backwards), while for the latter
there is only one link and the arrow only points from $\mu_{pt}$ to
$\lambda_{t}$ as no feedback is allowed.

The model comparison is carried out in terms of bias, root mean square error
(RMSE), 95% credible interval (CI) coverage and 95% CI width.

###  3.2 Simulation results

Table 2 presents the results of the simulation study in terms of the indexes
above. It is clear that the hierarchical two-component model framework
(H2Mjoint / H2M) outperforms the model that uses measured air pollution
concentration (ME, as in (3.1)); across the six pollutants the bias is reduced
by 3 to 11 fold and the coverage of the CI95% is always above 90% (compared to
53-77% for the ME model). In terms of precision, the RMSE is generally smaller
for the Bayesian model, indicating better accuracy in the estimates, while the
width of the confidence interval is larger, which can be explained by the
additional uncertainty included in the concentration estimates and that feeds
forward into the health component. This can also be seen in the 95% CI plot
for the $\bm{\beta}$ coefficients (Figure 2). Comparing H2Mjoint with H2M
shows that there is an advantage in allowing for the joint specification of
the two model components: in H2Mjoint the bias is smaller, which is clearer
when the true effects are different from 0 ($\beta_{1}$-$\beta_{3}$); at the
same time there is no increase in the estimate uncertainty, as the widths of
the 95% credible intervals do not change substantially.

Table 2: Results of the simulation study: the table shows the bias, root mean square error (RMSE), 95% credible intervals (CI) width and coverage for the ME, H2M and H2Mjoint. The bias and RMSE are substantially reduced for all the 6 pollutant coefficients using H2Mjoint / H2M. Coverage improves and at the same time width of the 95% credible interval increases, suggesting that the uncertainty is larger for the hierarchical two-component modelling framework, as expected, given that this comes also from the pollutant component. The comparison of H2Mjoint with H2M shows how the influence of the outcome helps reduce the bias, while at the same time the uncertainty does not increase. | Bias | RMSE
---|---|---
| ME | H2M | H2Mjoint | ME | H2M | H2Mjoint
$\beta_{1}$ | -0.021 | -0.007 | -0.002 | 0.003 | 0.002 | 0.002
$\beta_{2}$ | -0.036 | -0.006 | 0.002 | 0.004 | 0.005 | 0.004
$\beta_{3}$ | 0.013 | 0.004 | 0.000 | 0.003 | 0.004 | 0.002
$\beta_{4}$ | 0.008 | 0.003 | 0.002 | 0.001 | 0.002 | 0.001
$\beta_{5}$ | 0.021 | 0.002 | -0.001 | 0.002 | 0.002 | 0.002
$\beta_{6}$ | 0.022 | 0.002 | -0.001 | 0.002 | 0.002 | 0.002
| 95% CI width | 95% CI coverage
| ME | H2M | H2Mjoint | ME | H2M | H2Mjoint
$\beta_{1}$ | 0.16 | 0.20 | 0.20 | 65 | 92 | 93
$\beta_{2}$ | 0.20 | 0.30 | 0.30 | 53 | 97 | 97
$\beta_{3}$ | 0.16 | 0.17 | 0.16 | 71 | 92 | 97
$\beta_{4}$ | 0.13 | 0.16 | 0.16 | 77 | 98 | 99
$\beta_{5}$ | 0.16 | 0.19 | 0.22 | 61 | 95 | 94
$\beta_{6}$ | 0.15 | 0.18 | 0.19 | 65 | 97 | 99

##  4 Real application results

We focused on H2Mjoint to evaluate the effects of the six pollutants (CO, NO2,
O3, SO2, PM2.5 and PCNT) on daily CVD mortality in Greater London for
2011-2012, as the simulation showed that allowing for the feedback from the
outcome leads to an improvement in the estimates in terms of bias. In addition
to the multi pollutant H2Mjoint, we ran single pollutant models as a
comparison, given that this is the typical approach in the field. The DIC was
smaller for the model with 6 knots on time and 3 on temperature and humidity
(9257), thus we present the results of this model specification (which was
also the one used in Atkinson et al.2). For the models with increased number
of knots and different prior the results are presented in Table 1 and 2 of
Supplementary material, together with their DIC.

The model is able to reproduce the temporal pattern seen in the data, as shown
in the residuals (Figure 5 of Supplementary Material), that are scattered
around 0 for all the six pollutants as well as for the mortality counts.

As the pollutants are on different scales, to make their effects comparable,
in Table 3 we present the results in terms of percent increase for a
interquartile range (IQR) change in air pollution concentrations, defined as:

$\%\text{increase}=(e^{\beta_{p}\times IQR}-1)\times 100\%$

Table 3: Posterior mean and 95% credible interval of the percent increase in mortality for an IQR change in pollutant concentration: (left) multi pollutant H2Mjoint model; (centre) single pollutant H2Mjoint model; (right) single pollutant frequentist model (Atkinson et al., 2016). Note that all the pollutants are measured in $\mu g/m^{3}$ except for PCNT which is measured in $p/mm^{3}$.  |  | Multi Pollutants | Single Pollutants | Single Pollutants
---|---|---|---|---
|  | (H2Mjoint) | (H2Mjoint) | (Atkinson et al., 2016)
Pollutant | IQR | % Increase | % Increase | % Increase
|  | (95% CI) | (95% CI) | (95% CI)
CO | 0.10 | -1.67 | (-4.72, 1.65) | -1.59 | (-3.89, 0.84) | * | *
NO2 | 23.65 | 9.40 | (3.06, 16.03) | -0.25 | (-2.90, 2.43) | -1.69 | (-3.97, 0.64)
O3 | 26.85 | 3.46 | (0.18, 6.71) | 2.61 | (0.02, 5.32) | 3.31 | ( 0.83, 5.84)
SO2 | 2.20 | -1.94 | (-6.59, 2.80) | -1.13 | (-4.96, 3.15) | -2.33 | (-4.18, -0.45)
PM2.5 | 8.00 | -1.24 | (-3.45, 0.92) | -0.79 | (-2.06, 0.47) | -0.9 | (-2.09, 0.25)
PCNT | 5.18 | -2.89 | (-6.36, 1.05) | -0.31 | (-3.56, 3.35) | * | *
∗ CO and PCNT were not analysed in Atkinson et al., 2016.

Out of the six metrics NO2 and O3 shows an increased risk of CVD mortality,
with credible intervals entirely above 0, suggesting strong evidence of an
effect. For the remaining pollutants the point estimates of percent change are
slightly below 0, but there is a high degree of uncertainty on the results and
the credible intervals include 0. There is high correlation between measured
and latent pollutant concentration and, as expected, the latter is slightly
less extreme due to shrinkage intrinsic in the modelling framework (see figure
6 in Supplementary material for a plot comparing the posterior mean of
$\mu_{pt}$ with the measured concentration $Y_{pt}$ for the six pollution
metrics). In addition, the measurement error variance on the standardised
metrics is presented on Table 4 and shows the lowest values for NO2 and O3,
both around 0.04, while it is 0.08 for PM2.5 and it increases between 0.14 to
0.45 for the remaining pollutants. This suggests that the model is able to
account almost entirely for the variability of NO2, O3 and PM2.5, while it
would potentially point towards some residual confounding for SO2, CO and
PCNT.

By contrast the time-series single pollutant model shows a positive posterior
mean with a credible interval above zero only for O3. For the remaining
pollutants the point estimates are negative and their intervals cross 0,
pointing towards lack of substantive evidence of an effect. Measurement error
variances are larger, with a posterior mean spanning from 0.11 to 0.66,
suggesting that the multi pollutant model borrow strength across pollutants to
improve the accuracy of the concentration estimates.

The single pollutant results are in line with Atkinson et al.2, who analysed
the same period for Greater London and are reported on the right hand side of
Table 3. There is slightly more uncertainty in the H2Mjoint framework, as
expected, as the pollutant component contributes to it. This translates into
point estimates which are generally closer to zero and wider credible
intervals; it is particularly interesting to note how accounting for
uncertainty shift SO2 estimates towards zero, so that the protective effect
seen in Atkinson et al.2 disappears.

Table 4: Posterior Mean and 95% credible interval for the measurement error variance $\sigma^{2}_{v}$ for the H2Mjoint framework: multi pollutant model (left) and single pollutant model (right). Note that all the pollutants are measured in $\mu g/m^{3}$ except for PCNT which is measured in $p/mm^{3}$.  | Multi Pollutant Model | Single Pollutant Model
---|---|---
Pollutant | Posterior Mean (95% CI) | Posterior Mean (95% CI)
CO | 0.18 | (0.15, 0.22) | 0.44 | (0.36, 0.52)
NO2 | 0.03 | (0.01, 0.05) | 0.20 | (0.17, 0.27)
O3 | 0.04 | (0.02, 0.06) | 0.16 | (0.12, 0.21)
SO2 | 0.45 | (0.29, 0.51) | 0.66 | (0.55, 0.79)
PM2.5 | 0.08 | (0.04, 0.12) | 0.11 | (0.04, 0.18)
PCNT | 0.14 | (0.10, 0.17) | 0.59 | (0.49, 0.68)

##  5 Discussion

In this paper we proposed a fully Bayesian hierarchical model to assess the
health effect of multi pollutant concentrations in a time-series perspective,
allowing for the integration of uncertainty on the exposure and health
components. We deal with the common issue of multi-collinearity among
pollutants as the joint hierarchical specification allows to i) directly
estimate and incorporate the correlation when modelling the ‘true‘ latent
concentration from the measured ones at the monitoring site; ii) incorporate
such correlation in the link between concentration and health. The use of a
hierarchical model has been shown to provide stable estimates 23 and it allows
to specify an informative prior, which acts as a constraint on the parameter
estimates, helping deal with the potential colinearity among the pollutants.
In addition, a Bayesian approach naturally accounts for missing data in the
estimation process; this means that we were able to use the entire 731 days of
the time-series, while other analyses 2 were based on less data points as the
days with missing pollutant concentrations were removed. In our case, as we
are considering six pollutants at the same time, this would mean removing 125
days as one or more pollutants did not have concentration value recorded.

Note that the meteorological covariates (temperature and humidity) are
included in the exposure component as well as in the health component of the
model. This was done according to the approach proposed by Cefalu et al. 24
who discussed how the covariates included in the exposure prediction model
need to be included as confounders in the epidemiological model to avoid
biased results.

A key characteristic of our modelling framework is that it involves a joint
specification of the two components (pollution model and health model). In
this way all the uncertainty is accounted for in the estimation process,
differently from classical two-stage models, where the pollution concentration
estimates are considered without the associated uncertainty to evaluate their
health effects. At the same time, in the simulation study, we found that the
feedback from the outcome provides additional information to estimate the
health effects when these are truly different from 0, while at the same time
is not introducing bias in any direction when the true health effects are
null.

Running our model on simulated data we found that the proposed framework
caters well for the measurement error intrinsic in the observed concentrations
and is able to estimate the health effects more accurately than the model
which considers the observed concentrations as exposure in the health model.

At the same time, on the real data application, we showed that our multi
pollutant model is able to capture the short-term harmful effect of a possible
synergic mechanism between NO2 and O3. After adjustment for airborne particles
and other regulated gases (i.e. CO and SO2), we found a positive association
between a mixture of these two oxidant gases and cardiovascular mortality. The
result is in line with Williams et al.25, which considered a two-pollutant
model of NO2 and O3, but for daily counts of all cause mortality for Greater
London in 2000-2005, hence characterised by more power due to the longer
period and larger numbers. The plausibility of the reported associations is
also consistent with atmospheric chemistry findings 26 and toxicological
results 27. Williams et al.25 have looked into combining the two pollutants,
due to their high correlation and their complex chemistry; for instance NO2 is
a precursor of O3, but also scavenging it, which explains why in the centre of
cities the level of NO2 is higher and the level of O3 are lower. However a
clear drawback would be not to be able to disentangle the effects of the two
pollutants, which might act on health through different mechanisms. For this
reason we think that our approach is beneficial here, as it has shown the
ability to identify and quantify the magnitude of the short-term health effect
of the simultaneous exposure to multiple air pollutants, that it is not
detectable using a traditional single pollutant model 28. Therefore, from an
air quality management perspective we believe that a multi pollutant approach,
such as the one proposed in this study, has a potential for suggesting
effective control strategies to reduce adverse effects on human health, since
it is able to provide insights on the complex trade-offs between different
ambient pollutants.

In this paper we showed how the Bayesian hierarchical modelling framework is
advantageous for dealing with multi pollutant concentrations in a purely time-
series perspective. A natural extension will consists of increasing the number
of measurement sites, moving to a spatio-temporal model. This would allow to
account for natural spatial variation which can be particularly strong for
some of the pollutants (e.g. NO2) and should help increase the accuracy of the
pollution estimates, hence reducing the measurement error variance. In
addition such extension would allow to also evaluate chronic effects, which
are generally dominated by spatial heterogeneity.

## Acknowledgements

We thank Margaret Douglass for the health data extraction. The work of the UK
Small Area Health Statistics Unit (SAHSU) is funded by Public Health England
as part of the MRC-PHE Centre for Environment and Health, funded also by the
UK Medical Research Council. This paper does not necessarily reflect the views
of Public Health England or the Department of Health. SAHSU has ethics and
governance approvals to hold and use mortality data from the National Research
Ethics Service - reference 7/LO/0846 and from the Health Research Authority
Confidentially Advisory Group, reference HRA - 24/CAG/1039. We would also like
to thank colleagues within the Environmental Research Group at King’s College
London and at the National Physical Laboratory for providing the pollutant
measurements. MB and GF acknowledge partial support through the NERC/MRC
funded project: NE/I00789X/1 “Traffic Pollution and health in London”. MP and
MB acknowledge partial support through the MRC funded project: MR/M025195/1 “A
general framework to adjust for missing confounders in observational studies”.

All authors declare no conflict of interests.

Figure 1: Graphical representation of the modelling frameworks: (a) shows the
proposed two component model: the left hand side represents the pollutant
component, while the right hand side the health component. The latent
concentration for each pollutant and day, $\mu_{pt}$, obtained from the
pollutant component enters the health model as predictor. The specification of
the link between $\mu_{pt}$ and $\lambda_{t}$ makes the difference between H2M
and H2Mjoint. In the former the uncertainty from $\mu_{pt}$ goes forward into
the health model, but there is no feedback from $O_{t}$; in the latter the
uncertainty goes forward, while at the same time information from the
mortality count $O_{t}$ can influence back $\mu_{pt}$. (b) shows the ME model:
the pollutant component is not there and the measured pollutant concentration
$Y_{pt}$ is now directly linked to $\lambda_{t}$. For both (a) and (b) the
circles denote latent random variables, while the rectangles are observed
quantities; single rectangles are random variables, while double rectangles
enter the model as data and are not characterised by a probability
distribution. Figure 2: 95% posterior credible intervals for $\bm{\beta}$
under ME, H2M and H2Mjoint. The H2Ms show smaller levels of uncertainty, as
this influence the coefficients from the pollutant estimates as well as from
the health model itself. At the same time the ME model shows a larger bias in
the estimates, due to the measurement error, while H2Mjoint model show a
median estimate virtually equal to the true values, showing how the fedeback
from the outcome can play a role in reducing the corresponding bias.

## References

  * 1 W Xie, G Li, D Zao, X Xie, Z Wei, W Wang, M Wang, et al.  Relationship between fine particulate air pollution and ischaemic heart disease morbidity and mortality.  Heart, 101:257–263, 2014.
  * 2 R Atkinson, A Analitis, E Samoli, G Fuller, D Green, I Mudway, H Anderson, et al.  Short-term exposure to traffic-related air pollution and daily mortality in london, uk.  J Expo Sci Environ Epidemiol, 26:125–132, 2016.
  * 3 E Samoli, R Atkinson, A Analitis, G Fuller, D Green, I Mudway, H Anderson, et al.  Associations of short-term exposure to traffic-related air pollution with cardiovascular and respiratory hospital admissions in london, uk.  Occup. Environ. Med, 75:300–307, 2016.
  * 4 L Dai, A Zanobetti, P Koutrakis, and J Schwartz.  Associations of fine particulate matter species with mortality in the united states: A multicity time-series analysis.  Environ Health Perspect, 122:837–842, 2014.
  * 5 M Pirani, N Best, M Blangiardo, S Liverani, R Atkinson, and GW Fuller.  Analysing the health effects of simultaneous exposure to physical and chemical properties of airborne particles.  Environ Int, 79:56–64, 2015.
  * 6 J Bobb, L Valeri, B Henn, D Christiani, R Wright, M Mazumdar, J Godleski, et al.  Bayesian kernel machine regression for estimating the health effects of multi-pollutant mixtures.  Biostatistics, 16:493–508, 2015.
  * 7 G Smith, Z Bawa, Y Macklin, R Morbey, A Dobney, S Vordoulakis, and A Elliot.  Using real-time syndromic surveillance systems to help explore the acute impact of the air pollution incident of march/april 2014 in england.  Environ Res, 136:500–504, 2015.
  * 8 F Finazzi, M Scott, and A Fassó.  A model-based framework for air quality indices and population risk evaluation, with an application to the analysis of scottish air quality data.  J R Stat Soc Ser C, 62:287–308, 2013.
  * 9 G Huang, M Scott, and D Lee.  Multivariate space-time modelling of multiple air pollutants and their health effects accounting for exposure uncertainty.  Stat Med, pages 1–15, 2017.
  * 10 RW Atkinson, GW Fuller, RH Anderson, RM Harrison, and B Armstrong.  Urban ambient particle metrics and health: a time-series analysis.  Epidemiology, 21:501–511, 2010.
  * 11 DC Beddows, RM Harrison, DC Green, and GW Fuller.  Receptor modelling of both particle composition and size distribution from a background site in london, UK.  Atmospheric Chem Phys, 15:10107–10125, 2015.
  * 12 A Font and GW Fuller.  Did policies to abate atmospheric emissions from traffic have a positive effect in London?  Environ Pollut, 218:463–474, 2016.
  * 13 WHO Europe.  Review of evidence on health aspects of air pollution.  WHO Regional Office for Europe, Copenhagen, Denmark, 2013.
  * 14 EO Owens, MM Patel, E Kirrane, TC Long, J Brown, I Cote, MA Ross, and others.  Framework for assessing causality of air pollution-related health effects for reviews of the national ambient air quality standards.  Regul Toxicol Pharmacol., 88:332–337, 2017.
  * 15 IC Mills, RW Atkinson, HR Anderson, RL Maynard, and DP Strachan.  Distinguishing the associations between daily mortality and hospital admissions and nitrogen dioxide from those of particulate matter: a systematic review and meta-analysis.  BMJ Open, 6(7), 2016.
  * 16 G Shaddick and J Wakefield.  Modelling daily multivariate pollutant data at multiple sites.  J R Stat Soc Ser C, 51:351–372, 2002.
  * 17 D Ruppert, MP Wand, and RJ Carroll.  Semiparametric Regression.  Cambridge University Press, 2003.
  * 18 C Crainiceanu, D Rupert, and MP Wand.  Bayesian analysis for penalized spline regression using winbugs.  J Stat Softw, 14:1–24, 2005.
  * 19 D Lunn, C Jackson, N Best, A Thomas, and D Spiegelhalter.  The BUGS Book.  Chapman & Hall/CRC, 2012.
  * 20 F Dominici, JM Samet, and SL Zeger.  Combining evidence on air pollution and daily mortality from the 20 largest us cities: a hierarchical modelling strategy.  J R Stat Soc Ser A, 163:263–284, 2000.
  * 21 DJ Spiegelhalter, NG Best, BP Carlin, and A Van Der Linde.  Bayesian measures of model complexity and fit.  J R Stat Soc Ser B, 64(4):583–639, 2002.
  * 22 R Peng, F Dominici, and T Louis.  Model choice in time series studies of air pollution and mortality.  J R Stat Soc Ser A, 169(2):179–203, 2006.
  * 23 R MacLehose, D Dunson, A Herring, and J Hoppin.  Bayesian methods for highly correlated exposure data.  Epidemiology, 18:199–207, 2007.
  * 24 M Cefalu and F Dominici.  Does exposure prediction bias health effect estimation? the relationship between confounding adjustment and exposure prediction.  Epidemiology, 25:583–590, 2014.
  * 25 M Williams, R Atkinson, R Anderson, and F Kelly.  Associations between daily mortality in london and combined oxidant capacity, ozone and nitrogen dioxide.  Air Qual Atmos Health, 7:407–414, 2014.
  * 26 LF Gamon, JM White, and U Wille.  Oxidative damage of aromatic dipeptides by the environmental oxidants no${}_{2}^{\ast}$ and o3.  Org Biomol Chem, 12:8280–8287, 2014.
  * 27 DM Drechsler-Parks.  Cardiac output effects of o3 and no2 exposure in healthy older adults.  Toxicol Ind Health, 11:99–109, 1995.
  * 28 Joe L Mauderly and Jonathan M Samet.  Is there evidence for synergy among air pollutants in causing health effects?  Environ Health Perspect, 117(1):1, 2009.