# Resilience through Scene Context in Visual Referring Expression Generation

Simeon Junker  Sina Zarrieß
Bielefeld University
<EMAIL_ADDRESS>

###### Abstract

Scene context is well known to facilitate humans’ perception of visible
objects. In this paper, we investigate the role of context in Referring
Expression Generation (REG) for objects in images, where existing research has
often focused on distractor contexts that exert pressure on the generator. We
take a new perspective on scene context in REG and hypothesize that contextual
information can be conceived of as a resource that makes REG models more
resilient and facilitates the generation of object descriptions, and object
types in particular. We train and test Transformer-based REG models with
target representations that have been artificially obscured with noise to
varying degrees. We evaluate how properties of the models’ visual context
affect their processing and performance. Our results show that even simple
scene contexts make models surprisingly resilient to perturbations, to the
extent that they can identify referent types even when visual information
about the target is completely missing.

##  1 Introduction

Objects do not appear randomly in the world that surrounds us, but they occur
in predictable spatial, semantic, or functional configurations and relations
to their environment. Research on human perception shows that we “see the
world in scenes” (Bar, 2004), and that prior experience and knowledge of the
world helps us to efficiently process visual stimuli. Even with an extremely
short glimpse at an image, humans remember essential semantic aspects of the
scene and object arrangement (Oliva and Torralba, 2006). This rapid scene
understanding allows us to handle the complexity of the visual world and to
recognize objects in context, e.g., when they are not fully visible (Võ,
2021).

In this paper, we take a new perspective on how visual context facilitates the
automatic generation of descriptions for visual objects, a task well-known as
Referring Expression Generation (REG).

TRFv-0.0 | couch on right (A)
---|---
TRFv-0.5 | right brown chair (F)
TRFv-1.0 | right couch (A)
TRFt-0.0 | right couch (A)
TRFt-0.5 | right couch (A)
TRFt-1.0 | right elephant (F)
TRFs-0.0 | couch on right (A)
TRFs-0.5 | right couch (A)
TRFs-1.0 | right couch (A)

Figure 1:  Example from RefCOCO testB (displayed with noise level $0.5$) with
generated expressions and human judgements. Scene context enables target
identification even with full occlusion (TRFv-1.0, TRFs-1.0)

In past years, datasets have become available that provide referring
expressions for objects in images, with objects appearing in relatively
complex real-world contexts (Kazemzadeh et al., 2014; Mao et al., 2016). Yet,
recent work in this area has largely followed the traditional REG paradigm by
Dale and Reiter 1995, where (visual) context is mainly considered in terms of
so-called distractor objects, that are similar to the target and must
therefore be excluded by naming differences (Krahmer and van Deemter, 2012).

These distractors do not facilitate the description task, but even exert
“contextual pressure”, as the speaker needs to reason about which attributes
and words make the expression unambiguous (Cohn-Gordon et al., 2018; Schüz and
Zarrieß, 2021). The main goal of this paper is to widen this commonly accepted
view on the role of visual context in visual REG and investigate how
contextual information can be conceived as a resource that makes the
generation of descriptions easier rather than harder.

In visual REG from images, scene and object information is not available a
priori: Whereas classical REG algorithms mostly rely on symbolic scene
representations, neural generation models in visual REG have to extract object
properties from low-level visual representations of the target and its context
(Schüz et al., 2023). This even applies to properties as fundamental as the
type of an object, i.e. how it is named in the expression. Under ideal
conditions, determining a referent’s type and properties can be regarded as a
relatively simple task, but it becomes non-trivial in the presence of
imperfect visual information, occlusion or noise. At the same time, global
visual scene context can be expected to be of great support in this task, in
light of previous findings on human scene understanding (cf. Section 2).
However, to date, little is known as to how processes of scene understanding
and object type identification interact in REG.

In this work, we hypothesize that visual scene context makes REG models more
resilient, i.e., it allows them to recalibrate predictions that were based on
imperfect target representations. To test this, we adopt a novel experimental
setup for REG: we train and test different model architectures with target
representations that have been artificially obscured with varying degrees of
noise (cf. Figure 1). We provide the models with different context
representations and compare their performance concerning common quality
metrics and a focused human evaluation of their ability to determine referent
types. We test how certain properties of the visual context affect the
processing and performance of REG models, and verify our results with
experiments using further datasets that are substantially different from the
ones commonly used in existing REG research. Our results show that context
makes models surprisingly resilient to perturbations in target
representations, to the extent that they can identify referent types even when
information about the objects themselves is completely missing. We believe
that these results offer new perspectives on the role of scene context in
visual REG.

##  2 Background

#### Human scene understanding

Research on human vision and perception emphasizes the fact that scenes are
not mere collections of objects (Võ, 2021). When humans view a scene, they do
not simply recognize the objects in it, but they understand it is a coherent
whole. Oliva and Torralba (2006) observe that humans perceive the so-called
gist of a scene rapidly and even when local information is missing (e.g.
blurred). Experiments in this field indicate that contextual information can
facilitate the recognition of visible objects across different tasks (Oliva
and Torralba, 2007; Divvala et al., 2009; Galleguillos and Belongie, 2010;
Parikh et al., 2012), and that, on the other hand, incongruent context can
also be misleading (Zhang et al., 2020; Gupta et al., 2022). This means that
the human vision exploits learned knowledge about these regularities of the
visual word for visual processing (Biederman, 1972; Bar, 2004; Greene, 2013;
Pereira and Castelhano, 2014; Sadeghi et al., 2015). To model these
regularities, Võ (2021) proposed the notion of a “scene grammar” that can
account for the interaction of global and local visual perception and
understanding in humans.

#### Scenes, objects, and image captioning

Much research on V&L is currently concerned with modeling the generation and
understanding of image descriptions, e.g. in tasks like image captioning or
retrieval. Yet, many captioning tasks focus on rather object-centric
descriptions that mention objects and their spatial relationships (Cafagna et
al., 2021). A common representation of scene context in image captioning is
scene graphs, cf. (Yang et al., 2023), which are usually modeled via spatial
relations between bounding boxes of objects. Cafagna et al. 2023 propose a new
task and dataset that foregrounds scene-level instead of object-centric
descriptions. Another perspective on scene knowledge in captioning models is
coming from work that focuses on probing them with perturbed or systematically
varied images: Yin and Ordonez (2017) found that captioning with extremely
reduced inputs of labeled object layouts performs surprisingly well. Related
to this, Nikolaus et al. (2019) find that image captioning models often rely
on regularities in object occurrences, to the extent that they fail to
generalize to new combinations of objects. Their solution is to generate
unseen combinations and challenge models on these. Our goal in this work is
complementary: we aim to understand how exactly generation models may be able
to leverage regular scene knowledge and patterns of object co-occurrence, and
how this may facilitate the handling of imperfect visual information.

#### REG and scene context

REG is concerned with the generation of descriptions that distinguish a
particular object in a given visual context, cf. Krahmer and van Deemter 2012.
Recent visual REG models usually build on image captioning models but are
adapted to generate more pragmatically informative expressions, using e.g.
training objectives (Mao et al., 2016), comprehension modules (Luo and
Shakhnarovich, 2017), reinforcement agents (Yu et al., 2017) or decoding
strategies (Schüz and Zarrieß, 2021). REG models usually process different
forms of context information. Whereas some models encode differences in
appearance between targets and surrounding objects (Yu et al., 2016, 2017;
Tanaka et al., 2019; Kim et al., 2020; Liu et al., 2020), others use
representations of the global image (Mao et al., 2016; Luo and Shakhnarovich,
2017; Zarrieß and Schlangen, 2018; Panagiaris et al., 2020, 2021). Visual
context is often supplemented with the relative position and size of the
target in the image (Mao et al., 2016; Yu et al., 2017; Luo and Shakhnarovich,
2017; Li and Jiang, 2018; Tanaka et al., 2019; Kim et al., 2020; Panagiaris et
al., 2020; Liu et al., 2020).

#### Research gap

Little is known about how visual REG models internally exploit their context
representations and in what way context exactly enhances the generation of
expressions. Here, the implicit assumption often is that models exploit
context in a similar way as symbolic REG models, e.g. the Incremental
Algorithms by Reiter and Dale (2000). However, a key difference to symbolic
REG is that in visual REG failures in the scene and object understanding can
arise from model hallucination or imperfect visual input, cf. (Schüz et al.,
2023). This is especially evident for the type of objects: this attribute had
a privileged role in early works (Dale and Reiter, 1995) as they are essential
as the heads of referential noun phrases. In visual REG, referents must first
be correctly identified to name them appropriately (Zarrieß and Schlangen,
2017; Silberer et al., 2020a, b), which is challenging in cases of deficient
input, e.g. small or partially occluded objects (Yao and Fei-Fei, 2010). In
this paper, we aim to close this gap and investigate how visual context
information helps REG models to be more resilient to deficits in their target
inputs.

##  3 Experimental Set-Up

###  3.1 Outline and Research Hypotheses

The main idea of this work is to train and test standard REG models on visual
target representations occluded with varying amounts of noise, to investigate
how different combinations of target and context can compensate for this
perturbation. For this, we draw on existing model architectures, and evaluate
the trained models using both out-of-the-box quality metrics and more fine-
grained human evaluation capturing the validity of assigned referent type
labels. The evaluation results are also supported by supplementary analyses as
well as further experiments with an additional data set.

Generally, we expect that automatic metrics and human evaluation scores will
drop for increasing amounts of target noise. However, we also hypothesize that
visual context makes models more resilient, i.e., for the same amount of
noise, models supplied with context outperform variants with only target
information. While we expect this general effect across all conditions, it
should be more pronounced as the amount of occlusion increases.

###  3.2 Models

We set up two transformer-based REG models: TRF and CC. TRF is a transformer
trained from scratch on REG data, CC builds upon a pre-trained language model.
We define variants of both models using a) different combinations of target
and context representations, as the respective model inputs, and b) the amount
of target noise during training and inference. Implementation and training
details for our models can be found in appendix B.

Target representations include the visual contents of the target bounding box
($V_{t}$), its location, and size relative to the global image ($Loc_{t}$). As
context representations, we use the embedding of the global image with the
target masked out ($V_{c}$). For TRF, which is our better performing model, we
also experiment with scene-level information (or scene summaries) about what
kinds of objects are present in the surrounding scene ($S_{c}$), which are
derived from panoptic segmentations (Kirillov et al. 2018, see Section 3.3).
Models processing only target information are indicated with the subscript
$t$, whereas models processing $V_{c}$ and $S_{c}$ context information are
indexed with $v$ and $s$, respectively.

To test our systems for perturbed target representations, we randomly replaced
a fixed proportion of the pixels in the bounding box contents with random
noise during both training and inference. All systems are trained and tested
with three noise settings: $0.0$ as our baseline setting, where no pixels are
perturbed; $0.5$, where $50\%$ of the pixels are replaced with noise; and
$1.0$, where the entire content of the target bounding box is occluded, i.e.
no visual information about the target is available. Noise levels for training
and evaluation are shown in the index of the model identifiers.

#### Vanilla Transformer (TRF)

We use the model from Schüz and Zarrieß (2023), which is based on an existing
implementation for image captioning.111https://github.com/saahiluppal/catr The
model builds on ResNet (He et al., 2015) encodings for targets and context,
which are passed on to an encoder/decoder transformer in the style of Vaswani
et al. (2017). The model is largely comparable to the system in Panagiaris et
al. (2021), but without self-critical sequence training and layer-wise
connections between encoder and decoder. Unlike e.g. Mao et al. (2016), we
train the model using Cross Entropy Loss.

We compare three variants of this model, which take as input concatenated
feature vectors comprised of the representations described above. TRFt
receives only target information, i.e. the input vector is composed as
$[V_{t};Loc_{t}]$. In addition to this, TRFv receives representations of the
global image, with the input vector structure $[V_{t};Loc_{t};V_{c}]$.
Finally, TRFs takes scene-level representations about the relative area
occupied by different object classes in the visual context, i.e.
$[V_{t};Loc_{t};S_{c}]$.

For both $V_{t}$ and $V_{c}$, the respective parts of the image are scaled to
$224\times 224$ resolution (keeping the original ratio and masking out the
padding) and encoded with ResNet-152 (He et al., 2015), resulting in
representations with 196 features ($14\times 14$) and hidden size $512$ for
both target and context. $Loc_{t}$ is a vector of length 5 with the corner
coordinates of the target bounding box and its area relative to the whole
image, projected to the model’s hidden size. The scene summary input for TRFs
consists of 134 features, which represent the relative area all of the 134
object or stuff types in COCO occupy in the visual context. To use this
information in our model, we embed each of the object and stuff types in an
additional embedding layer with $512$, which is jointly trained with the
model. In the model’s forward pass, we weight each of the individual category
embeddings with the relative area it occupies in the respective input image,
and form $S_{c}$ by concatenating the embeddings.

#### Fine-tuned GPT-2 (CC)

We adapt the ClipCap model in Mokady et al. (2021) to the REG task. The
authors use a simple MLP-based mapping network to construct fixed-size
prefixes for GPT-2 (Radford et al., 2019) from CLIP encodings (Radford et al.,
2021), and fine-tune both the mapping network and the language model for the
image captioning task. To the best of our knowledge, this is the first model
tested for REG which utilizes a pre-trained language model.

As for the TRF model, we compare different variants of this base architecture.
First, in CCt, the GPT-2 prefix is constructed as $[V_{t};Loc_{t}]$, where
$V_{t}$ is computed like the CLIP prefix in the original paper (but for the
contents of the target bounding box) and $Loc_{t}$ is the location features
described above, projected into a single prefix token. For CCv, we again add
the global image (minus the target) as context: $V_{c}$ is computed in the
same way as $V_{t}$, but with a separate mapping network and with the global
image as the visual input. Here, the final prefix is $[V_{t};V_{c};Loc_{t}]$.

###  3.3 Data

We use RefCOCO (Kazemzadeh et al., 2014) for training and evaluation, which
contains bounding boxes and expressions for MSCOCO images (Lin et al., 2014).
The dataset contains separate testA and testB splits (1.9k and 1.8k items),
where testA only contains humans as referential targets and testB all other
object classes (but not humans).

We also conduct experiments on the detection dataset PACO-EGO4D (Ramanathan et
al., 2023), which contains annotations for objects and object parts in first-
person video frames (Grauman et al., 2022). In comparison to RefCOCO, PACO is
larger (75k items in test split), data is less standardized and objects are
often harder to recognize.

To construct the scene summaries for our TRFs model and analyze attention
allocation patterns, we rely on annotations for panoptic segmentation
(Kirillov et al., 2018), i.e. dense pixel-level segmentation masks for both
thing and stuff classes in MSCOCO images (Caesar et al., 2016).

###  3.4 Evaluation

#### Generation Quality / N-Gram Metrics

To estimate the general generation capabilities of our models we rely on BLEU
(Papineni et al., 2002) and CIDEr (Vedantam et al., 2014) as established
metrics for automatic evaluation.

#### Referent Type Assignment / Human Evaluation

To test whether our models succeed in assigning valid types to referents, we
collect human judgements for generated expressions for a subset of 200 items
from the RefCOCO testB split. This split contains only non-human referents,
and was chosen due to preliminary tests indicating that judgments about the
validity of type labels for human referents are often difficult, e.g. due to
ambiguity regarding the gender of depicted individuals. The annotators were
instructed to rate only those parts of the expressions that refer to the type
of the referential target. For example, “the black dog” should be rated as
correct if the target is of the type dog, but is actually white. All items
should be assigned exactly one of the following categories:

  * •

Adequate / A: The generated expression contains a valid type description for
the referent.

  * •

Misaligned / M: Type designators do not apply to the intended target, but to
other objects (partially) captured by the bounding box.

  * •

Omission / O: Omission of the target type, e.g. description via non-type
attributes, pronominalization or general nouns such as “thing”.

  * •

False / F: Type designations that do not apply to the intended target or other
objects captured by the bounding box.

Previous research has shown considerable variation in object naming (Silberer
et al. 2020a, b, among others). Therefore, for the A category, type
descriptions do not have to match the ground truth annotations, but different
labels can be considered adequate if they represent valid descriptions of the
target type. For example, dog, pet and animal would be considered equally
correct for depicted dogs. Subsequent to the human evaluation, we investigate
correlations between the evaluation results and further properties of the
visual context.

#### Referent Type Assignment / Classification Accuracy

We complement the human evaluation with RefCOCO with further experiments using
PACO-EGO4D. While PACO does not provide referring expressions, we treat the
object and object-part annotations similarly, i.e. our models generate the
category strings (instead of predicting the respective category in a
multiclass classification scheme). We evaluate the identification of object
and part types by measuring the accuracy of the models in exactly reproducing
the respective category strings (for entire objects) or the strings in the
object-part tuples (for object parts).

#### Attention Allocation

We also examine how our TRFv model allocates attention over different parts of
the input as a result of different noise levels during training. First, we
follow Schüz and Zarrieß (2023) in measuring the ratio between target and
context partitions, i.e. the summed attention weights directed to the target
and its context in both the encoder and decoder multi-head attention. For
this, we compute $\alpha_{t}$, $\alpha_{l}$ and $\alpha_{c}$ as the cumulative
attention weights directed to $V_{t}$, $Loc_{t}$ and $V_{c}$, respectively, by
calculating the sum of the attention weights assigned to each input partition,
normalized such that $\alpha_{t}+\alpha_{l}+\alpha_{c}=1$. We also quantify
the attention difference between $\alpha_{t}$ and $\alpha_{c}$ as
$\Delta_{t,c}$, by excluding $\alpha_{l}$ and normalizing the target and
context scores such that $\alpha_{t}+\alpha_{c}=1$. Then, we calculate
$\Delta_{t,c}=\alpha_{t}-\alpha_{c}$, i.e. $0<\Delta_{t,c}\leq 1$ if there is
relative focus on the target, $-1\leq\Delta_{t,c}<0$ if there is relative
focus on the context, and $\Delta_{t,c}=0$ when both parts are weighted
equally.

Second, we measure the model attention allocated to different classes of
objects in the visual context, using the panoptic segmentation data described
in Section 3.3. Here, we first interpolate the model attention map to fit the
original dimensions of the image, and retrieve the segmentation masks for the
respective image. For each category $x\in X$, we then compute the cumulative
attention weight $\alpha_{x}$ by computing the sum of pixels attributed to
this category, weighted by the model attention scores over the image and
normalized such that $\sum_{x\in X}\alpha_{x}=1$. We report $\alpha_{x=tgt}$,
i.e. attention allocated to areas of the visual context assigned the same
category as the referential target. As the covered area varies between object
categories, we get different scores even if the model attention is perfectly
balanced over the image. To address this, we also report scores that are
normalized by the area covered by the category. Scores $>1$ indicate that the
category is attended more than to be expected based on the coverage area.

##  4 Results on RefCOCO

###  4.1 Automatic Quality Metrics

| testA | testB
---|---|---
| Bl1 | Bl2 | CDr | Bl1 | Bl2 | CDr
system |  |  |  |  |  |
TRFt-0.0 | 0.55 | 0.35 | 0.86 | 0.57 | 0.35 | 1.28
TRFt-0.5 | 0.49 | 0.32 | 0.73 | 0.51 | 0.32 | 1.04
TRFt-1.0 | 0.35 | 0.17 | 0.34 | 0.30 | 0.14 | 0.20
TRFv-0.0 | 0.58 | 0.39 | 0.93 | 0.61 | 0.39 | 1.36
TRFv-0.5 | 0.54 | 0.35 | 0.81 | 0.56 | 0.36 | 1.24
TRFv-1.0 | 0.46 | 0.29 | 0.60 | 0.55 | 0.36 | 1.14
TRFs-0.0 | 0.54 | 0.34 | 0.84 | 0.57 | 0.35 | 1.27
TRFs-0.5 | 0.52 | 0.35 | 0.81 | 0.56 | 0.35 | 1.28
TRFs-1.0 | 0.42 | 0.24 | 0.51 | 0.53 | 0.33 | 1.12
CCt-0.0 | 0.48 | 0.30 | 0.70 | 0.47 | 0.28 | 0.88
CCt-0.5 | 0.38 | 0.22 | 0.48 | 0.36 | 0.20 | 0.52
CCt-1.0 | 0.35 | 0.16 | 0.37 | 0.29 | 0.12 | 0.16
CCv-0.0 | 0.57 | 0.38 | 0.92 | 0.58 | 0.37 | 1.25
CCv-0.5 | 0.51 | 0.32 | 0.77 | 0.49 | 0.31 | 0.97
CCv-1.0 | 0.40 | 0.23 | 0.46 | 0.38 | 0.21 | 0.46
Table 1:  BLEU1, BLEU2 and CIDEr scores on RefCOCO testA and testB for all TRF
and CC variants. Systems indicated with t can only access target information,
v and s models are supplied with visual context and scene summaries,
respectively. Target noise proportions ($0.0$, $0.5$, $1.0$) are denoted in
the indices. Generally, context information leads to improved results,
especially for high noise settings.

Table 1 shows the results of the automatic evaluation of our systems on
RefCOCO testA and testB. Interestingly, throughout all conditions, the simpler
TRF model outperforms CC, although the latter builds on pre-trained CLIP and
GPT-2 which are known to be effective for image captioning (Mokady et al.,
2021). It is possible that CC cannot fully benefit from CLIP pre-training due
to the structural differences between bounding box contents and full images,
or that performance drops result from higher compression when constructing the
GPT prefixes. Also, TRF achieves a considerably larger performance gain than
CC when adding scene context, indicating that this model is more effective at
exploiting contextual information.

For both TRF and CC, scores consistently drop with increasing target noise.
However, this is mitigated if context is available: For both model types,
variants incorporating visual context are substantially more robust against
target noise, even in cases where target representations are entirely occluded
by noise ($1.0$ in the subscripts). A striking example is RefCOCO testB, where
CIDEr drops to $0.20$ for TRFt-1.0 and $0.16$ for CCt-1.0, but TRFv-1.0
achieves scores as high as $1.14$. Here, CCv-1.0 drastically underperforms
with CIDEr $0.46$, but still outperforms its no-context counterpart.

Interestingly, we see considerable differences between testA and testB. Both
TRFt-1.0 and CCt-1.0 achieve better results on testA, but the scores are
generally higher on testB. Importantly, testA is restricted to human
referents, while testB encompasses all non-human object classes. Therefore,
models without any access to meaningful visual input could often guess right
on the frequent human classes, but struggle with the higher variation in
testB. This is supported by the inverse pattern that visual context
particularly improves the testB results. Here, differences between $t$ and $v$
variants are much higher, suggesting that context is more informative for non-
human objects, i.e. there are stronger associations between certain types of
objects and the contexts in which they occur.

Another striking result is that the same patterns emerge if we exchange visual
context representations with more abstract scene summaries: TRFs-1.0 achieves
CIDEr $1.12$ for entirely occluded targets in testB, comparable to TRFv-1.0.
Interestingly, between TRFs-0.0 and TRFs-0.5 the scene model slightly improves
in CIDEr scores, i.e. it can fully compensate for partial target occlusion.

###  4.2 Target Identification

Human judgements were collected from 5 expert annotators, including the first
author. Every system was evaluated independently by three annotators, with a
Fleiss’ Kappa of 0.85, indicating almost perfect agreement (Landis and Koch,
1977). The final judgments are determined by majority vote.

The human evaluation results for the 200-item subset of RefCOCO testB are
shown in Table 2 and illustrated in Figure 2. Generally, the results mirror
the pattern in the BLEU and CIDEr scores discussed previously: Across all
conditions, A scores drop if noise ratios increase, while F scores increase at
the same time. For M and O the results are less clear, but higher noise
settings generally lead to higher rates than the baseline setting for both
categories. This holds for TRF and CC, but TRF again performs better
throughout all conditions. Again, visual context clearly allows the models to
compensate for deficient target representations: While CCv-1.0 assigns
adequate types in almost $20\%$ of all cases (as compared to $0.5\%$ without
context information), TRFv-1.0 scores an impressive $66\%$, only $15.5\%$ less
than without any target noise.

Interestingly, scene summaries allow the model to compensate for deficient
target representations even more effectively. Across all noise settings, TRFs
achieves the highest A scores and the lowest F and O rates, even without any
target noise, unlike for BLEU and CIDEr (cf. Section 4.1).

| % A | % F | % O | % M
---|---|---|---|---
TRFt-0.0 | 84.0 | 10.5 | 5.0 | 0.5
TRFt-0.5 | 66.0 | 27.5 | 4.0 | 2.5
TRFt-1.0 | 1.5 | 75.5 | 19.5 | 3.5
TRFv-0.0 | 81.5 | 12.0 | 5.5 | 1.0
TRFv-0.5 | 70.5 | 18.5 | 7.0 | 4.0
TRFv-1.0 | 66.0 | 26.5 | 4.0 | 3.5
TRFs-0.0 | 89.0 | 7.0 | 3.5 | 0.5
TRFs-0.5 | 81.0 | 14.5 | 2.5 | 2.0
TRFs-1.0 | 68.0 | 22.0 | 1.5 | 8.5
CCn-0.0 | 45.5 | 46.5 | 7.0 | 1.0
CCn-0.5 | 23.0 | 61.0 | 13.0 | 3.0
CCn-1.0 | 0.5 | 84.5 | 11.0 | 4.0
CCv-0.0 | 76.0 | 21.0 | 3.0 | 0.0
CCv-0.5 | 55.0 | 36.0 | 6.5 | 2.5
CCv-1.0 | 19.5 | 68.5 | 9.0 | 3.0
$human$ | 91.0 | 2.0 | 6.5 | 0.5
Table 2: Results of the human evaluation on 200 items from RefCOCO-testB.
Generally, contextual information leads to more adequate type descriptions,
even if target representations are entirely occluded.

###  4.3 How do models exploit scene context?

So far, our results indicate that the scene context of referential targets
greatly improves the resilience of REG, to the extent that correct predictions
are possible to a surprising rate even if target information is missing. Here,
we aim to analyze how exactly contextual information is exploited by the
models. As discussed in Section 2, previous research indicates that
regularities of object co-occurrence and scene properties facilitate e.g.
object recognition in context. However, qualitative inspection of our data
indicates that for high noise, our systems often copy from context, i.e.
correctly predict referent types that are also present in the surrounding
scene, given that many classes of objects tend to appear in groups. We
investigate this in more detail and (a) perform statistical tests to check
whether similar objects in context support identification performance and (b)
analyze the attention distribution for TRFv to see whether the model learns to
attend to the respective objects in context.

#### Statistical analysis: Target categories in context

We hypothesize that recalibration through context should be more effective
when the target class is also present in the scene. To test this, we conduct a
correlation analysis between identification accuracy and the relative coverage
of the target class in the context. For this, we again rely on panoptic
segmentation annotations (cf. Section 3.3) to compute the proportion of pixels
of the same class as the referential target, normalized by the total size of
the context. We binarize the human evaluation scores (True if rated as A, else
False) and compute the Point-biserial correlation coefficient between the
relative coverage of the target class in context and the identification
accuracy.

For both TRFv-1.0 (corr: 0.321, p < 0.001) and TRFs-1.0 (corr: 0.277, p <
0.001) we found that a higher prevalence of the target class in the visual
context leads to significantly higher scores in human evaluation, i.e. systems
can easier compensate a lack of visual target information if the context
contains similar objects. For CCv-1.0, we see the same trend, but without
statistical significance (corr: 0.136, p = 0.055). For cases, where neither
system can identify the target class, we see a strong inverse correlation
(corr: -0.267, p < 0.001).

#### Model attention to target category in context

| Encoder | Decoder
---|---|---
| $\alpha_{x=tgt}$ | norm. | $\alpha_{t}$ | $\alpha_{l}$ | $\alpha_{c}$ | $\Delta_{t,c}$ | $\alpha_{x=tgt}$ | norm. | $\alpha_{t}$ | $\alpha_{l}$ | $\alpha_{c}$ | $\Delta_{t,c}$
TRFv-0.0 | 36.70 | 1.77 | 44.49 | 9.20 | 46.31 | -0.02 | 26.94 | 1.20 | 52.65 | 9.69 | 37.67 | 0.16
TRFv-0.5 | 35.27 | 1.64 | 18.90 | 16.06 | 65.04 | -0.55 | 40.56 | 2.05 | 32.65 | 14.41 | 52.94 | -0.24
TRFv-1.0 | 35.63 | 1.70 | 41.05 | 0.67 | 58.28 | -0.17 | 43.66 | 2.26 | 43.75 | 0.48 | 55.78 | -0.12
Table 3: Attention allocation scores for TRFv, averaged over all RefCOCO testB
items. $\alpha$ scores are reported in %.

To see whether the TRFv model has indeed learned the hypothesized copying
strategy, we compute the distribution of attention mass directed to target,
location and context partitions as well as to objects sharing the target
category in the visual context, as described in Section 3.4.

In Table 3, we report the analysis results, averaged for all items in the
RefCOCO testB split. The $\Delta_{t,c}$ scores show that the context partition
receives more attention if the target is occluded with noise during the
training, in line with our previous results. However, surprisingly, more
attention is allocated to the context in the $0.5$ noise setting than if no
target information is accessible. The $\alpha$ scores also indicate that
location features are especially focused in this case, suggesting that this
source of information is especially helpful if visual target information is
reduced, but not entirely missing.

As shown by the $\alpha_{x=tgt}$ scores in Table 3, target noise during model
training does not seem to have a consistent effect on encoder attention to
context objects sharing the target category. For the decoder, however, we see
a notable increase: Whereas the baseline model assigns an average of 26.94 %
of its attention mass on context objects with the target class, this is
significantly increased for higher noise settings (40.56 % and 43.66 %). The
normalized results exhibit the same patterns, i.e. as a result of target
noise, context objects sharing the target class receive more than double of
the attention mass as to be expected based on their size in the image.

These results suggest that the TRF model learns to exploit the occurrence of
similar objects in target and context as a common property of scenes in
RefCOCO. However, due to the prevalence of frequent object classes and the
reliance on published photos, it is unclear how representative these results
are. In the next section, we examine whether these patterns can be replicated
for the PACO dataset.

##  5 Results on PACO

In our experiments on the EGO4D portion of the PACO dataset (Section 3.3), we
treat the category strings in the detection dataset as expressions and train
TRFt and TRFc to generate those strings given the contents of the target
bounding box and (for the latter variant) the visual context (see Section
3.4). We report accuracy scores for the test split in Table 4. Here, notably,
the TRFt variant achieves higher accuracy scores than TRFv, unless the entire
visual target representation is covered with random noise. This suggests that
visual context is less informative or more difficult to process in PACO than
RefCOCO. However, the (comparably small) gain of TRFv-1.0 over TRFt-1.0
indicates that the model can leverage the visual context to a certain degree.
While some of the differences to RefCOCO may result from different
experimental settings (e.g. class strings instead of expressions), the PACO
results also hint towards general problems with datasets relying on scraped
images such as RefCOCO, in that they may not be not sufficiently
representative of the visual complexity in everyday scenes.

| $acc_{obj}$ | $acc_{obj-part}$
---|---|---
TRFt-0.0 | 60.93 | 34.03
TRFt-0.5 | 45.30 | 25.57
TRFt-1.0 | 16.47 | 7.55
TRFv-0.0 | 56.97 | 33.22
TRFv-0.5 | 34.92 | 20.31
TRFv-1.0 | 22.34 | 11.11
Table 4: Results for TRFt and TRFv on PACO-EGO4D. $acc_{obj}$ describes the
accuracy for reproducing object category strings, $acc_{obj-part}$ for
reproducing (object, part) tuples for annotated object parts.

##  6 Discussion and Conclusion

Our findings show that contextual information about the surroundings of
referential targets makes REG models more resilient against pertubations in
visual target representations. Even for conditions where no target information
is present at all, REG models maintain good results in automatic quality
metrics and identify referent types with high accuracy, as shown in the human
evaluation results. This holds for different kinds of context: While
especially the TRFv model is able to leverage ResNet encodings of image
contents outside the target bounding box, the same applies to scene-level
representations of depiced objects, as included in the TRFs model.

Interestingly, our subsequent analysis suggest that our context models
implicitly learned to copy from the visual context, i.e. assign labels to
referents which also apply to visible context objects. While the weaker
context effects in our PACO results suggest that this strategy is not
universally applicable, it appears to be highly effective the more regular
RefCOCO data. This is in stark contrast to basic assumptions of the REG
paradigm, where context information is considered important mainly to ensure
that references can be resolved without ambiguity. Here, we show, that is also
a valuable source for further communicative goals, i.e. the truthfulness of
generated expressions.

Overall, our results indicate that the influence of visual context in REG is
more multifaceted than reflected in previous studies. Importantly, however,
this study only provides an initial spotlight, as research in related fields
suggests that there are other and more complex ways in which visual scene
context may facilitate reference production. With this in mind, we strongly
advocate further research into scene context at the interface of perceptual
psychology, computer vision and language generation.

##  7 Limitations

We identify the following limitations in our study:

First, in both training and evaluation, we do not consider pragmatic
informativeness as a core criterion for the REG task. We train our models
using Cross Entropy Loss and do not test whether the generated expressions
unambiguously describe the referential target, instead focusing on semantic
adequacy as an important prerequisite for the generation of successful
referential expressions. However, we acknowledge that a comprehensive view
would require the consideration of both semantic and pragmatic aspects.

Also, we do not consider recent developments such as multimodal LLMs, although
the high diversity of their training data would contribute an interesting
aspect to this study. Here, we selected our models with a focus on both
modifiability and transparent processing.

Finally, additional vision and language datasets such as VisualGenome (Krishna
et al., 2016) would have made the results more representative. However, due to
time and space constraints, we leave this for future research.

## References

  * Bar (2004) Moshe Bar. 2004.  Visual objects in context.  _Nature Reviews Neuroscience_ , 5(8):617–629.
  * Biederman (1972) Irving Biederman. 1972.  Perceiving real-world scenes.  _Science_ , 177(4043):77–80.
  * Caesar et al. (2016) Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2016.  Coco-stuff: Thing and stuff classes in context.
  * Cafagna et al. (2021) Michele Cafagna, Kees van Deemter, and Albert Gatt. 2021.  What vision-language models ‘see’ when they see scenes.
  * Cafagna et al. (2023) Michele Cafagna, Kees van Deemter, and Albert Gatt. 2023.  HL dataset: Visually-grounded description of scenes, actions and rationales.  In _Proceedings of the 16th International Natural Language Generation Conference_ , pages 293–312, Prague, Czechia. Association for Computational Linguistics.
  * Cohn-Gordon et al. (2018) Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018.  Pragmatically informative image captioning with character-level inference.
  * Dale and Reiter (1995) Robert Dale and Ehud Reiter. 1995.  Computational interpretations of the gricean maxims in the generation of referring expressions.  _Cognitive Science_ , 19(2):233–263.
  * Divvala et al. (2009) Santosh K. Divvala, Derek Hoiem, James H. Hays, Alexei A. Efros, and Martial Hebert. 2009.  An empirical study of context in object detection.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE.
  * Galleguillos and Belongie (2010) Carolina Galleguillos and Serge Belongie. 2010.  Context based object categorization: A critical survey.  _Computer Vision and Image Understanding_ , 114(6):712–722.
  * Grauman et al. (2022) Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * Greene (2013) Michelle R. Greene. 2013.  Statistics of high-level scene context.  _Frontiers in Psychology_ , 4.
  * Gupta et al. (2022) Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille. 2022.  Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015.  Deep residual learning for image recognition.
  * Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014.  ReferItGame: Referring to objects in photographs of natural scenes.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 787–798, Doha, Qatar. Association for Computational Linguistics.
  * Kim et al. (2020) Jungjun Kim, Hanbin Ko, and Jialin Wu. 2020.  CoNAN: A complementary neighboring-based attention network for referring expression generation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1952–1962, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kirillov et al. (2018) Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. 2018.  Panoptic segmentation.
  * Krahmer and van Deemter (2012) Emiel Krahmer and Kees van Deemter. 2012.  Computational generation of referring expressions: A survey.  _Computational Linguistics_ , 38(1):173–218.
  * Krishna et al. (2016) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.
  * Landis and Koch (1977) J. Richard Landis and Gary G. Koch. 1977.  The measurement of observer agreement for categorical data.  _Biometrics_ , 33(1):159.
  * Li and Jiang (2018) Xiangyang Li and Shuqiang Jiang. 2018.  Bundled object context for referring expressions.  _IEEE Transactions on Multimedia_ , 20(10):2749–2760.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * Liu et al. (2020) Jingyu Liu, Wei Wang, Liang Wang, and Ming-Hsuan Yang. 2020.  Attribute-guided attention for referring expression generation and comprehension.  _IEEE Transactions on Image Processing_ , 29:5244–5258.
  * Luo and Shakhnarovich (2017) R. Luo and Gregory Shakhnarovich. 2017.  Comprehension-guided referring expressions.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3125–3134.
  * Mao et al. (2016) Junhua Mao, J. Huang, A. Toshev, Oana-Maria Camburu, A. Yuille, and Kevin Murphy. 2016.  Generation and comprehension of unambiguous object descriptions.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11–20.
  * Mokady et al. (2021) Ron Mokady, Amir Hertz, and Amit H. Bermano. 2021.  Clipcap: Clip prefix for image captioning.
  * Nikolaus et al. (2019) Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliott. 2019.  Compositional generalization in image captioning.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 87–98, Hong Kong, China. Association for Computational Linguistics.
  * Oliva and Torralba (2006) Aude Oliva and Antonio Torralba. 2006.  Chapter 2 building the gist of a scene: the role of global image features in recognition.  In _Progress in Brain Research_ , pages 23–36. Elsevier.
  * Oliva and Torralba (2007) Aude Oliva and Antonio Torralba. 2007.  The role of context in object recognition.  _Trends in Cognitive Sciences_ , 11(12):520–527.
  * Panagiaris et al. (2020) Nikolaos Panagiaris, Emma Hart, and Dimitra Gkatzia. 2020.  Improving the naturalness and diversity of referring expression generation models using minimum risk training.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 41–51, Dublin, Ireland. Association for Computational Linguistics.
  * Panagiaris et al. (2021) Nikolaos Panagiaris, Emma Hart, and Dimitra Gkatzia. 2021.  Generating unambiguous and diverse referring expressions.  _Computer Speech & Language_, 68:101184.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Parikh et al. (2012) Devi Parikh, C. Lawrence Zitnick, and Tsuhan Chen. 2012.  Exploring tiny images: The roles of appearance and contextual information for machine and human object recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 34(10):1978–1991.
  * Pedregosa et al. (2011) Fabian Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pereira and Castelhano (2014) Effie J. Pereira and Monica S. Castelhano. 2014.  Peripheral guidance in scenes: The interaction of scene context and object content. _Journal of Experimental Psychology: Human Perception and Performance_ , 40(5):2056–2072.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ramanathan et al. (2023) Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. 2023.  Paco: Parts and attributes of common objects.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * Reiter and Dale (2000) Ehud Reiter and Robert Dale. 2000.  _Building natural language generation systems_.  Cambridge University Press, Cambridge, U.K. New York.
  * Sadeghi et al. (2015) Zahra Sadeghi, James L. McClelland, and Paul Hoffman. 2015.  You shall know an object by the company it keeps: An investigation of semantic representations derived from object co-occurrence in visual scenes.  _Neuropsychologia_ , 76:52–61.
  * Schüz and Zarrieß (2021) Simeon Schüz and Sina Zarrieß. 2021.  Decoupling pragmatics: Discriminative decoding for referring expression generation.  In _Proceedings of the Reasoning and Interaction Conference (ReInAct 2021)_ , pages 47–52, Gothenburg, Sweden. Association for Computational Linguistics.
  * Schüz and Zarrieß (2023) Simeon Schüz and Sina Zarrieß. 2023.  Keeping an eye on context: Attention allocation over input partitions in referring expression generation.  In _Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023)_ , pages 20–27, Prague, Czech Republic. Association for Computational Linguistics.
  * Schüz et al. (2023) Simeon Schüz, Albert Gatt, and Sina Zarrieß. 2023.  Rethinking symbolic and visual context in referring expression generation.  _Frontiers in Artificial Intelligence_ , 6.
  * Silberer et al. (2020a) Carina Silberer, Sina Zarrieß, and Gemma Boleda. 2020a.  Object naming in language and vision: A survey and a new dataset.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 5792–5801, Marseille, France. European Language Resources Association.
  * Silberer et al. (2020b) Carina Silberer, Sina Zarrieß, Matthijs Westera, and Gemma Boleda. 2020b.  Humans meet models on object naming: A new dataset and analysis.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1893–1905, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Tanaka et al. (2019) M. Tanaka, Takayuki Itamochi, K. Narioka, Ikuro Sato, Y. Ushiku, and T. Harada. 2019\.  Generating easy-to-understand referring expressions for target identifications.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5793–5802.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Vedantam et al. (2014) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014.  Cider: Consensus-based image description evaluation.
  * Võ (2021) Melissa Le-Hoa Võ. 2021.  The meaning and structure of scenes.  _Vision Research_ , 181:10–20.
  * Yang et al. (2023) Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, and Yu Zhang. 2023.  Transforming visual scene graphs to image captions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12427–12440, Toronto, Canada. Association for Computational Linguistics.
  * Yao and Fei-Fei (2010) Bangpeng Yao and Li Fei-Fei. 2010.  Modeling mutual context of object and human pose in human-object interaction activities.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE.
  * Yin and Ordonez (2017) Xuwang Yin and Vicente Ordonez. 2017.  Obj2Text: Generating visually descriptive language from object layouts.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 177–187, Copenhagen, Denmark. Association for Computational Linguistics.
  * Yu et al. (2016) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016\.  Modeling context in referring expressions.  In _Computer Vision – ECCV 2016_ , pages 69–85, Cham. Springer International Publishing.
  * Yu et al. (2017) Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017.  A joint speaker-listener-reinforcer model for referring expressions.  In _Computer Vision and Pattern Recognition (CVPR)_ , volume 2.
  * Zarrieß and Schlangen (2017) Sina Zarrieß and David Schlangen. 2017.  Obtaining referential word meanings from visual and distributional information: Experiments on object naming.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 243–254, Vancouver, Canada. Association for Computational Linguistics.
  * Zarrieß and Schlangen (2018) Sina Zarrieß and David Schlangen. 2018.  Decoding strategies for neural referring expression generation.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 503–512, Tilburg University, The Netherlands. Association for Computational Linguistics.
  * Zhang et al. (2020) Mengmi Zhang, Claire Tseng, and Gabriel Kreiman. 2020.  Putting visual object recognition in context.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12982–12991.

##  Appendix A Risks and Ethical Considerations

We do not believe that there are significant risks associated with this work,
as we consider the generation of general expressions for generic objects in
freely available datasets with limited scale. When selecting samples for human
evaluation, we refrain from descriptions of people (that could potentially be
perceived as hurtful). No ethics review was required. Our data (published upon
acceptance) does not contain any protected information and will be fully
anonymized prior to publication.

##  Appendix B Model implementation and training

For the hyperparameters of our models, we largely followed Panagiaris et al.
(2021) (TRF) and Mokady et al. (2021) (CC). During inference, we relied on
(deterministic) greedy decoding.

The TRF model has 3 encoder and 3 decoder layers with 8 attention heads,
hidden dimension and feedforward dimension of 512, and was trained with a
initial learning rate of 0.0001 for the transformer encoder and decoder, and
0.00001 for the pre-trained ResNet-152 backbone. Our TRF models have
approximately 103,000,000 parameters.

For our CC model, we kept the settings defined by Mokady et al. (2021). From
the two models proposed in this work, we used the variant where a simple MLP
is used as a mapping network and the GPT-2 language model is fine-tuned during
training. However, we have different prefix sizes than in the original paper:
For CCt, we have a prefix size of 11, i.e. 10 for the visual target
representation and 1 for the target location information. For CCv, our prefix
size is 21, with additional 10 tokens for the visual context representation.
The model was trained using a learning rate of 0.00001. CCv has approximately
338,000,000 and CCt has 307,000,000 parameters.

We trained our models on an Nvidia RTX A40. RefCOCO contains 42k and PACO-
EGO4D contains 116k items for training. The number of training epochs per
system and the final CIDEr scores over the validation sets are displayed in
Table 5. We trained all our models for a maximum of 15 epochs, with early
stopping if no new maximum for CIDEr over the validation set has been achieved
for three consecutive epochs. Per epoch, the compute time was approximately
2.30 h for TRF and CC on RefCOCO and 4.30 h for TRF on PACO.

| dataset | epochs | CIDEr (val)
---|---|---|---
TRF${}_{t}-0.0$ | RefCOCO | 8 | 1.074
TRF${}_{t}-0.5$ | RefCOCO | 11 | 0.936
TRF${}_{t}-1.0$ | RefCOCO | 5 | 0.302
TRF${}_{v}-0.0$ | RefCOCO | 6 | 1.156
TRF${}_{v}-0.5$ | RefCOCO | 9 | 1.035
TRF${}_{v}-1.0$ | RefCOCO | 6 | 0.869
TRF${}_{s}-0.0$ | RefCOCO | 8 | 1.075
TRF${}_{s}-0.5$ | RefCOCO | 14 | 1.032
TRF${}_{s}-1.0$ | RefCOCO | 12 | 0.818
CG${}_{t}-0.0$ | RefCOCO | 7 | 0.824
CG${}_{t}-0.5$ | RefCOCO | 8 | 0.554
CG${}_{t}-1.0$ | RefCOCO | 2 | 0.294
CG${}_{v}-0.0$ | RefCOCO | 4 | 1.103
CG${}_{v}-0.5$ | RefCOCO | 10 | 0.894
CG${}_{v}-1.0$ | RefCOCO | 7 | 0.526
TRF${}_{t}-0.0$ | PACO | 3 | 3.236
TRF${}_{t}-0.5$ | PACO | 8 | 2.662
TRF${}_{t}-1.0$ | PACO | 7 | 0.814
TRF${}_{v}-0.0$ | PACO | 3 | 3.554
TRF${}_{v}-0.5$ | PACO | 14 | 3.047
TRF${}_{v}-1.0$ | PACO | 5 | 2.15
Table 5: Training information for all TRF and CC variants.

##  Appendix C Scientific Artifacts

In our work, we mainly used scientific artifacts in the form of existing model
implementations, all of which are cited or referenced in Section 3. The model
implementations were published under permissive licences, i.e. MIT (TRF) and
Apache 2.0 (CC). Upon acceptance, we will publish our modifications to the
model implementations using the same licences, and our other code and data
using permissive licences. Apart from this, we relied on scikit-learn (version
1.2.0, Pedregosa et al. 2011) for our statistic analysis and the RefCOCO API
(Kazemzadeh et al., 2014; Yu et al.,
2016)222https://github.com/lichengunc/refer for computing BLEU and CIDEr
scores.

##  Appendix D Human Evaluation

We conducted a human evaluation in which the adequacy of assigned referent
types in English referring expressions was assessed. The annotation guidelines
are published as supplementary material. Our annotators were undergrad student
assistants from linguistics and computational linguistics, which were paid by
the hour according to the applicable pay scale. The annotators were informed
about the intended use of their produced data. Along with our code, upon
acceptance, we will publish the fully anonymized raw and aggregated results of
the human evaluation.

Figure 2 illustrates the results of the human evaluation described in Section
4.2.

Figure 2: Visualization of human evaluation results for all tested systems and
a sample of human annotations in RefCOCO testB.


#  _Ab initio_ design of charge-mismatched ferroelectric superlattices

Claudio Cazorla  Institut de Ci$\grave{e}$ncia de Materials de Barcelona
(ICMAB-CSIC), 08193 Bellaterra, Spain  Massimiliano Stengel  Institut de
Ci$\grave{e}$ncia de Materials de Barcelona (ICMAB-CSIC), 08193 Bellaterra,
Spain  ICREA - Institució Catalana de Recerca i Estudis Avançats, 08010
Barcelona, Spain<EMAIL_ADDRESS>
###### Abstract

We present a systematic approach to modeling the electrical and structural
properties of charge-mismatched superlattices from first principles. Our
strategy is based on bulk calculations of the parent compounds, which we
perform as a function of in-plane strain and out-of-plane electric
displacement field. The resulting two-dimensional phase diagrams allow us to
accurately predict, without performing further calculations, the behavior of a
layered heterostructure where the aforementioned building blocks are
electrostatically and elastically coupled, with an arbitrary choice of the
interface charge (originated from the polar discontinuity) and volume ratio.
By using the [PbTiO3]m/[BiFeO3]n system as test case, we demonstrate that
interface polarity has a dramatic impact on the ferroelectric behavior of the
superlattice, leading to the stabilization of otherwise inaccessible bulk
phases.

###### pacs:

71.15.-m, 77.22.Ej, 77.55.+f, 77.84.Dy

##  I Introduction

When layers of perovskite oxides are epitaxially stacked to form a
periodically repeated heterostructure, new intriguing functionalities can
emerge in the resulting superlattice [ghosez08, ; junquera11, ]. These are
further tunable via applied electric fields and thermodynamic conditions, and
thus attractive for nanoelectronics and energy applications. An excellent
example is the [PbTiO3]m/[SrTiO3]n system, where the polarization,
tetragonality, piezoelectric response, and ferroelectric transition
temperature strongly change with the volume ratio of the parent compounds
[dawber05, ; dawber07, ; dawber12, ]. Such a remarkable tunability is usually
rationalized in terms of epitaxial strains [dawber05b, ], electrostatic
coupling (see Fig. 1a) [zubko12, ; wu12, ], and local interface effects
[junquera12, ; bousquet08, ].

While perovskite titanates with ATiO3 formula (A=Sr, Pb, Ba or Ca) have
traditionally been the most popular choice as the basic building blocks, a
much wider range of materials (e.g., BiFeO3) is currently receiving increasing
attention by the community. The motivation for such an interest is clear: a
superlattice configuration provides the unique opportunity of enhancing
materials properties via “strain engineering”, and a multifunctional compound
such as BiFeO3 appears to be a natural candidate in this context. (For
example, strain has been predicted to enhance the magnetoelectric response of
BiFeO3 by several orders of magnitude compared to bulk samples [wojdel09, ;
wojdel10, ].) Also, a superlattice geometry can alleviate the leakage issues
of pure BiFeO3 films [ranjith07, ; ranjith08, ].

Combining a III–III perovskite like BiFeO3 (or I–V, like KNbO3) with a II–IV
titanate appears, however, problematic from the conceptual point of view. In
fact, the charge-family mismatch inevitably leads to polar (and hence
electrostatically unstable) interfaces between layers murray09 . This is not
necessarily a drawback, though: recent research has demonstrated that polar
interfaces can be, rather than a nuisance to be avoided, a rich playground to
be exploited for exploring exciting new phenomena. The prototypical example is
the LaAlO3/SrTiO3 system, where a metallic two-dimensional electron gas
appears at the heterojunction to avoid a “polar catastrophe” nakagawa06 ;
ohtomo04 . Remarkably, first-principles calculations have shown that
interfaces in oxide superlattices can remain insulating provided that the
layers are thin enough, and produce rather dramatic effects on the respective
polarization of the individual components bristowe09 ; murray09 . This means
that, in a superlattice, polar discontinuities need not be compensated by
electronic or ionic reconstructions; they can, instead, be used as an
additional, powerful materials-design tool to control the behavior of the
polar degrees of freedom therein. Such a control may be realized, for
instance, by altering the stoichiometry at the interfaces (see Fig. 1b). To
fully explore the potential that this additional degree of freedom (the
interface built-in polarity) provides, and guide the experimental search for
the most promising materials combinations, one clearly needs to establish a
general theoretical framework where the “compositional charge” murray09  is
adequately taken into account.

Figure 1: (Color online) (a) Description of the electrostatic coupling in a
ferroelectric (orange)/paraelectric (blue) bilayer; $P$, $\mathcal{E}$, and
$D$ represent the component of the polarization, electric field and electric
displacement vectors along the stacking direction, and $\sigma_{\rm int}$ is
the interface charge density. (b) Intermixed AO-type interfaces in a
[BiFeO3]m/[PbTiO3]n superlattice and the resulting interface charge densities.
(c) Illustration of the $20$-atom simulation cell used in our calculations;
red, blue and black spheres represent O, B, and A atoms in the ABO3
perovskite.

In this Letter, we present a general first-principles approach to predict the
behavior of charge-mismatched perovskite oxide superlattices based exclusively
on the properties of their individual bulk constituents. Our formalism
combines the constrained-$D$ strategies of Wu et al. wu08 , which are key to
decomposing the total energy of the system into the contributions of the
individual layers, with the rigorous description of the interface polarity
proposed in Ref. stengel11 . As a result, we are able to exactly describe the
electrostatic coupling and mechanical boundary conditions, enabling a clear
separation between genuine interface and bulk effects. Crucially, the present
method allows one to quantify, in a straightforward way, the impact that
interface polarity has on the equilibrium (and metastable) phases of the
superlattice. As a proof of concept we apply our formalism to the study of
[PbTiO3]m/[BiFeO3]n (PTO/BFO) heterostructures. We find that (i) our _bulk_
model accurately matches earlier first-principles predictions obtained for
ultrashort-period superlattices (i.e., $m=n=3$) by using explicit _supercell_
simulations stengel12 , and (ii) by assuming interface terminations with
different nominal charge, we obtain a radical change in the overall
ferroelectric properties of the superlattice, which demonstrates the crucial
role played by the polar mismatch.

Figure 2: (Color online) Energy of PTO/BFO superlattices with $a=3.81$ Å
expressed as a function of $D$, for selected values of $\lambda$ and
$\sigma_{\rm int}$. Equilibrium and metastable superlattice states are
represented with solid and empty dots. Red (green) vertical lines indicate
phase transitions occurring in bulk BFO (PTO) under different $D$ conditions.
(a) and (b) represent the cases of neutral and polar interfaces, respectively.

We start by expressing the total energy of a monodomain two-color superlattice
(i.e., composed of species A and B) as,

$U_{\rm tot}(D,\lambda,a)=\lambda\cdot U_{\rm
A}(D,a)+\left(1-\lambda\right)\cdot U_{\rm B}(D,a)~{}.$ (1)

Here $U_{\rm A}$ and $U_{\rm B}$ are the internal energies of the individual
constituents, $D$ is the electric displacement along the out-of-plane stacking
direction (i.e., $D\equiv{\cal E}+4\pi P$ where ${\cal E}$ is the electric
field and $P$ is the _effective_ polarization, relative to the centrosymmetric
reference configuration), $\lambda$ is the relative volume ratio of material A
(i.e, $\lambda\equiv m/(n+m)$ where $m$ and $n$ are the thicknessess of layers
A and B, respectively), and $a$ is the in-plane lattice parameter (we assume
heterostructures that are coherently strained to the substrate). Note that
short-range interface effects have been neglected. (While it is certainly
possible to incorporate the latter in the model, e.g. along the guidelines
described in Ref. wu08 , we believe these would have been an unnecessary
complication in the context of the present study.) By construction, Eq. (1)
implicitly enforces the continuity of $D$ along the out-of-plane stacking
direction (which we label as $z$ henceforth), which is appropriate for
superlattices where the interfaces are nominally uncharged [ghosez08, ;
junquera11, ].

In presence of a polar mismatch, one has a net “external” interface charge, of
compositional origin murray09 , $\sigma_{\rm int}$ (see Fig. 1a), which is
localized at the interlayer junctions. In such a case, Eq. (1) needs to be
revised as follows,

$U_{\rm tot}(D,\sigma_{\rm int},\lambda,a)=\lambda\cdot U_{\rm
A}(D,a)+\left(1-\lambda\right)\cdot U_{\rm B}(D-\sigma_{\rm int},a)~{},$ (2)

i.e. the $U_{\rm B}$ curve is shifted in $D$-space to account for the jump in
$D$ produced by $\sigma_{\rm int}$. (Recall the macroscopic Maxwell equation,
$\nabla\cdot{\bf D}=\rho_{\rm ext}$, where $\rho_{\rm ext}$, the “external”
charge, encompasses all contributions of neither dielectric nor ferroelectric
origin.) Once the functions $U_{\rm A}$ and $U_{\rm B}$ are computed and
stored (e.g. by using the methodology of Ref. stengel09b ), one can predict
the ground-state of a hypothetical A/B superlattice by simply finding the
global minimum of $U_{\rm tot}$ with respect to $D$ at fixed values of
$\sigma_{\rm int}$, $\lambda$ and $a$. The advantage of this procedure is
that, for a given choice of A and B, the aforementioned four-dimensional
parameter space can be explored very efficiently, as no further _ab initio_
calculations are needed.

It is useful, before going any further, to specify the physical origin of
$\sigma_{\rm int}$ in the context of this work. Consider, for example, a
periodic BiFeO3/PbTiO3 superlattice, which we assume (i) to be stoichiometric
(and therefore charge-neutral) as a whole, (ii) to have an ideal AO-BO2-AO-BO2
stacking along the (001) direction, and (iii) to form (say) AO-type interfaces
(see Fig. 1b). (The same arguments can be equally well applied to the case of
BO2-type interfaces.) Depending on the growth conditions, one can have a
certain degree of intermixing in the boundary AO layers, which will adopt an
intermediate composition Bix Pb(1-x)O. As a pure BiO layer is formally charged
$+1$ and PbO is neutral, we can readily write $\sigma_{\rm
int}=\pm\left(x-\frac{1}{2}\right)$ (expressed in units of $e/S$ with $S$
being the surface of the corresponding 5-atom perovskite cell), where the
choice of plus or minus depends on the arbitrary assignment of BiFeO3 and
PbTiO3 as the A or B component in Eq. (2) [see Fig. 1b]. In the following we
shall illustrate the crucial role played by $\sigma_{\rm int}$ (and hence, by
the interface stoichiometry) on the ferroelectric properties of a BFO/PTO
superlattice, by combining Eq. (2) with the bulk $U_{\rm BFO}(D,a)$ and
$U_{\rm PTO}(D,a)$ curves that we calculate from first principles.

Our calculations are performed with the “in-house” LAUTREC code within the
local spin density approximation to density-functional theory. (We
additionally apply a Hubbard $U=3.8$ eV to Fe ions kornev07 ; yang12 .) We use
the $20$-atom simulation cell depicted in Fig. 1c for both BFO and PTO, which
allows us to describe the ferroelectric and anti-ferrodistortive (AFD) modes
of interest (i.e. in-phase AFDzi and out-of-phase AFDzo and AFDxy, see Ref.
[bousquet08, ]). Atomic and cell relaxations are performed by constraining the
out-of-plane component of $D$ stengel09b  and the in-plane lattice constant
$a$ to a given value. [Calculations are repeated many times in order to span
the physically relevant two-dimensional $(D,a)$ parameter space.]

We start by illustrating the results obtained at fixed strain, $a=3.81$ Å (see
Fig. 2), by assuming $\sigma_{\rm int}=0$, which corresponds to fully
intermixed junctions ($x=0.5$), and we vary the BFO volume ratio, $\lambda$.
At the extreme values of $\lambda$, the results are consistent with the
expectations: the equilibrium configuration of BFO (i.e., the minimum of
$U_{tot}$ with $\lambda=1$) at this value of $a$ is the well-known R-type
$Cc$-I phase alison10 , derived from the bulk ground state via the application
of epitaxial compression; PTO ($\lambda=0$), on the other hand, is in a
tetragonal $P4mm$ phase with the polarization vector oriented out of plane.
Intermediate values of $\lambda$ yield a linear combination of the two single-
component $U(D)$ curves, where the spontaneous $P_{z}$ at equilibrium
gradually moves from the pure PTO to the pure BFO value.

Unfortunately, the possible equilibrium states that can be attained by solely
varying $\lambda$ (at this value of $a$ and $\sigma_{\rm int}$) lie far from
any physically “interesting” region of the phase diagram. For example, note
the kink at $|D|\sim$0.3 C/m2 in the pure BFO case, which corresponds to a
first-order transition to an orthorhombic $Pna2_{1}$ phase (a close relative
of the higher-symmetry $Pnma$ phase, occurring at $D=0$). A huge piezoelectric
and dielectric response is expected in BFO in a vicinity of the transition
cazorla14 , raising the question of whether one could approach this region by
playing with $\sigma_{\rm int}$, in addition to $\lambda$.

The answer is yes: when oxide superlattices with $\sigma_{\rm int}=0.5$ are
considered [corresponding to “ideal” (BiO)+/TiO2 and (FeO2)-/PbO interfaces],
the stable minimum of the system favors a smaller spontaneous polarization in
the BFO layers, approaching the aforementioned ($Cc{\rm-I}\to Pna2_{1}$) phase
boundary in the limit of small $\lambda$. Interestingly, the $U_{\rm tot}(D)$
curve becomes asymmetric (the interfacial charge breaks inversion symmetry),
and a secondary, metastable minimum appears. Overall, the resulting phase
diagram turns out to be much richer, with new combinations of phases emerging
(e.g. in region II’, where BFO exists in the orthorhombic $Pna2_{1}$ phase and
PTO in the tetragonal $P4mm$ phase), and highly non-trivial changes in the
electrical properties occurring as a function of $\lambda$.

Figure 3: Total energy (a) and out-of-plane electric displacement $D$ (b) of
the equilibrium (solid symbols) and metastable (empty symbols) states of
PTO/BFO superlattices with $\lambda=\frac{1}{2}$ and $\sigma_{\rm int}=0.5$,
expressed as a function of the in-plane lattice parameter. Regions in which
PTO and BFO exist in different phases are delimited with vertical dashed
lines; the corresponding space groups and AFD distortion patterns in Glazer’s
notation are shown in (a), and the components of the ferroelectric
polizarization in (b).

In order to further illustrate the power of our approach, we shall now fix the
volume ratio to $\lambda=0.5$ (corresponding to alternating BFO and PTO layers
of equal thickness) and vary the in-plane lattice parameter in the range
$3.6\leq a\leq 4.2$ Å . We shall first consider the case of charged interfaces
with $\sigma_{\rm int}=0.5$, as this choice allows for a direct comparison
with the results of Yang _et al._ (obtained via standard supercell
simulations) [stengel12, ]. In Fig. 3 we show the energy and spontaneous
electric displacement of the equilibrium and metastable states as a function
of $a$. Four regions can be identified in the diagrams depending on the phases
adopted by BFO and PTO at each value of the in-plane strain. (Their crystal
space groups, AFD pattern and in-plane / out-of-plane ferroelectric
polarization, respectively $P_{xy}$ and $P_{z}$, are specified in compact form
in the figure.) In region I’ both PTO and BFO adopt a monoclinic $Pc$ phase
characterized by large in-phase AFDz distortions and non-zero $P_{xy}$ and
$P_{z}$. Such a monoclinic $Pc$ phase is closely related to the orthorhombic
$Pmc2_{1}$ structure which has been recently predicted in PTO and BFO at large
tensile strains [yang12, ]. In region II’ PTO adopts an orthorhombic $Ima2$
phase, characterized by vanishing AFD distortions and a large in-plane ${\bf
P}$ (we neglect the small out-of-plane $P_{z}$), while BFO is in its well-
known $Cc$-I state. In region III, BFO remains $Cc$-I, while PTO adopts a
$P4mm$ phase, both with _opposite_ out-of-plane polarization with respect to
region II’. These structures switch back to a positively oriented $P_{z}$ in
region IV’, respectively transforming into a monoclinic $Cc$-II and a
tetragonal $I4cm$ phase. The $I4cm$ phase is characterized by anti-phase AFDz
distorsions and an out-of-plane ${\bf P}$, while the $Cc$-II corresponds to
the “supertetragonal” T-type phase of BFO zeches09 . Note that, as observed
already while discussing Fig. 2, the net interface charge leads to an
asymmetric double-well potential, and consequently to an energy difference
(typically of $\sim 20$ meV/f.u. or less, see Fig. 3a) between the two
oppositely polarized states. (Only one minimum survives at large tensile
strains, where the superlattice is no longer ferroelectric.) At the phase
boundaries such energy difference vanishes; the obvious kinks in the $U_{\rm
tot}$ curve shown in Fig. 3(a) indicate that the transitions (at $a=3.71$,
$3.87$, and $4.05$ Å) are all of first-order type.

The above results are in remarkable agreement with those of Yang _et al._
stengel12 . The only apparent discrepancy concerns the ordering of the
stable/metastable states in region III, which anyway involves a very subtle
energy difference (and is therefore sensitive to short-range interface
effects, not considered here). Obtaining such an accurate description of
superlattices where the individual layers are as thin as three perovskite
units stengel12  provides a stringent benchmark for our method, and validates
it as a reliable modeling tool. From the physical point of view this
comparison suggest that, even in the ultrathin limit, PTO/BFO superlattices
can be well understood in terms of macroscopic bulk effects, i.e., short-range
interface-specific phenomena appear to play a relatively minor role.

Figure 4: Same as in Fig. 3, but considering neutral interfaces. The out-of-
plane polarization is the same in PTO and BFO layers.

Having gained confidence in our method, we can use it to predict the behavior
of a hypothetical superlattice with $\sigma_{\rm int}=0$, corresponding to a
centrosymmetric reference structure with fully intermixed Pb0.5Bi0.5O
interface layers (see Fig. 4). Note the symmetry of the two opposite
polarization states, and the common value of the spontaneous electric
displacement adopted by BFO and PTO. The resulting phase diagram consists,
again, in four regions, with a first-order and two second-order phase
transitions occurring at $a=4.07$, $3.88$ and $3.73$ Å , respectively (see
Fig. 4a). In three of these regions, the individual layers display structures
which are different from those obtained in the $\sigma_{\rm int}=0.5$ case: in
region I both PTO and BFO stabilize in an orthorhombic $Pmc2_{1}$ phase
[yang12, ], characterized by a vanishing $P_{z}$; in region II PTO adopts a
monoclinic $Cm$ phase with the polarization roughly oriented along (111)
($P_{z}\neq P_{xy}\neq 0$) and no AFD, while BFO stabilizes in the already
discussed $Cc$-I phase; finally, in region IV, PTO is tetragonal $P4mm$ and
BFO is monoclinic $Cc$-II. These findings quantitatively demonstrates that the
interface charge mismatch can have a tremendous impact on the physical
properties of oxide superlattices. Our simple and general method allows one to
model and quantify accurately these effects, and most importantly to
rationalize them in terms of intuitive physical concepts.

In summary, we have discussed a general theoretical framework to predict the
behavior of charge-mismatched superlattices. We have showed that the effect of
the interface stoichiometry, which we describe via the “compositional”
interface charge $\sigma_{\rm int}$, is quite dramatic, and needs to be
properly accounted for in the models. More generally, we argue that
$\sigma_{\rm int}$ can be regarded, in addition to $\lambda$ and $a$, as a
further degree of freedom in designing oxide heterostructures with tailored
functionalities, opening exciting new avenues for future research.

###### Acknowledgements.

This work was supported by MICINN-Spain [Grants No. MAT2010-18113 and No.
CSD2007-00041], and the CSIC JAE-DOC program (C.C.). We thankfully acknowledge
the computer resources, technical expertise and assistance provided by RES and
CESGA.

## References

  * (1) P. Ghosez and J. Junquera, J. Comp. Theor. Nanosci. 5, 2071 (2008).
  * (2) C. Lichtensteiger et al., in Oxides Ultrathin Films: Science and Technology, edited by G. Pacchioni and S. Valeri, Ch. 12, 265 (Wiley-VCH, Germany, 2011).
  * (3) M. Dawber et al., Phys. Rev. Lett. 95, 177601 (2005).
  * (4) M. Dawber et al., Adv. Mater. 19, 4153 (2007).
  * (5) J. Sinsheimer et al., Phys. Rev. Lett. 109, 167601 (2012).
  * (6) M. Dawber, K. M. Rabe, and J. F. Scott, Rev. Mod. Phys. 77, 1083 (2005).
  * (7) P. Zubko et al., Nano Letters 12, 2846 (2012).
  * (8) C. W. Swartz and X. Wu, Phys. Rev. B 85, 054102 (2012).
  * (9) P. Aguado-Puente, P. García-Fernández, and J. Junquera, Phys. Rev. Lett. 107, 217601 (2011).
  * (10) E. Bousquet, M. Dawber, N. Stucki, C. Lichtensteiger, P. Hermet, S. Gariglio, J.-M. Triscone, and P. Ghosez, Nature (London) 452, 732 (2008).
  * (11) J. C. Wojdeł and J. ${\rm\acute{I}}$${\rm\tilde{n}}$iguez, Phys. Rev. Lett. 103, 267205 (2009).
  * (12) J. C. Wojdeł and J. ${\rm\acute{I}}$${\rm\tilde{n}}$iguez, Phys. Rev. Lett. 105, 037208 (2010).
  * (13) R. Ranjith, B. Kundys, and W. Prellier, Appl. Phys. Lett. 91, 222904 (2007).
  * (14) R. Ranjith et al., Appl. Phys. Lett. 92, 232905 (2008).
  * (15) E. D. Murray and D. Vanderbilt, Phys. Rev. B 79, 100102 (2009).
  * (16) N. Nakagawa, H. Y. Hwang, and D. A. Muller, Nature Mater. 5, 204 (2006).
  * (17) A. Ohtomo and H. Y. Hwang, Nature (London) 427, 423 (2004).
  * (18) N. C. Bristowe, E. Artacho, and P. B. Littlewood, Phys. Rev. B 80, 045425 (2009).
  * (19) X. Wu, M. Stengel, K. M. Rabe, and D. Vanderbilt, Phys. Rev. Lett. 101, 087601 (2008).
  * (20) M. Stengel and D. Vanderbilt, Phys. Rev. B 80, 241103(R) (2009).
  * (21) Y. Yang, M. Stengel, W. Ren, X. H. Yan, and L. Bellaiche, Phys. Rev. B 86, 144114 (2012).
  * (22) M. Stengel, N. A. Spaldin, and D. Vanderbilt, Nature Physics 5, 304 (2009).
  * (23) I. A. Kornev, S. Lisenkov, R. Haumont, B. Dkhil, and L. Bellaiche, Phys. Rev. Lett. 99, 227602 (2007).
  * (24) Y. Yang, W. Ren, M. Stengel, X. H. Yan, and L. Bellaiche, Phys. Rev. Lett. 109, 057602 (2012).
  * (25) A. J. Hatt, N. A. Spaldin, and C. Ederer, Phys. Rev. B 81, 054109 (2010).
  * (26) C. Cazorla and M. Stengel, to be published.
  * (27) R. J. Zeches et al., Science 326, 977 (2009).