#  Structured Generative Models for Scene Understanding

Christopher K. I. Williams
School of Informatics
University of Edinburgh, UK
<EMAIL_ADDRESS>

###### Abstract

This position paper argues for the use of _structured generative models_
(SGMs) for scene understanding. This requires the reconstruction of a 3D scene
from an input image, whereby the contents of the image are causally explained
in terms of models of instantiated objects, each with their own type, shape,
appearance and pose, along with global variables like scene lighting and
camera parameters. This approach also requires scene models which account for
the co-occurrences and inter-relationships of objects in a scene. The SGM
approach has the merits that it is compositional and generative, which lead to
interpretability.

To pursue the SGM agenda, we need models for objects and scenes, and
approaches to carry out inference. We first review models for objects, which
include “things” (object categories that have a well defined shape), and
“stuff” (categories which have amorphous spatial extent). We then move on to
review _scene models_ which describe the inter-relationships of objects.
Perhaps the most challenging problem for SGMs is _inference_ of the objects,
lighting and camera parameters, and scene inter-relationships from input
consisting of a single or multiple images. We conclude with a discussion of
issues that need addressing to advance the SGM agenda.

Keywords: structured generative models, generative models, compositionality,
scene understanding.

##  1 Introduction

The goal of this position paper is to promote the use of _structured
generative models_ (SGMs) for scene understanding. These models are situated
in the classical framework for computer vision whereby a 3D scene is
_reconstructed_ from one or more input images. In this case the contents of
the image are causally explained in terms of models of instantiated objects,
each with their own type, shape, appearance and pose, along with global
variables like scene lighting and camera parameters. This approach also
requires _scene models_ which account for the co-occurrences and inter-
relationships of objects in a scene. Because such models can _generate_ (or
reconstruct, or explain) the scene, and because they are _structured_ (i.e.
they are composed of multiple objects and their relationships), we term them
_structured generative models_.

This reconstructive framework is also known as _analysis-by-synthesis_
(Grenander,, 1978), or _vision-as-inverse-graphics (VIG)_ (Kulkarni et al.,,
2015; Moreno et al.,, 2016). It can be traced back to the early days of
“blocks world” research in the 1960s (Roberts,, 1963). Other early work in
this vein includes the VISIONS system of Hanson and Riseman, (1978), and the
system of Ohta et al., (1978) for outdoor scene understanding. For example,
the VISIONS system used various levels of analysis (e.g., objects, surfaces),
and mappings between the image-specific parse and generic knowledge about
scenes.

Alternatives to the VIG framework are either _discriminative approaches_ ,
predicting some target quantity or quantities given input image(s), or
_unstructured generative models_. Discriminative approaches are typically
applied to solve _specific tasks_ such as object detection or semantic
segmentation, which are usually specified in _image space_. These are usually
set up as supervised learning tasks, thus requiring annotated data. Currently
deep neural networks (DNNs) are the dominant method-of-choice for such tasks.
DNNs are often highly accurate, but as discriminatively-trained models they
can sometimes fail badly, producing absurd mistakes,111The phrase “absurd
mistakes” is borrowed from Daniel Kahneman’s talk at the NeurIPS conference in
December 2021. but with no effective indication of unreliability. One example
of this is performance failures on adversarial examples (see e.g., Szegedy et
al., 2013). Also as the discriminative models are trained on specific
datasets, they can often perform poorly when faced with the same task but on a
novel dataset with different statistics (distribution shift). The focus on the
evaluation of specific tasks means that the predictions from multiple tasks
are not required to create a coherent understanding of the input image in
terms of the 3D world; this point is made, e.g., by Zamir et al., (2020).

With _unstructured generative models_ , images are generated from a single set
of latent variables, without explicit modelling of objects and their
interactions. An example is the work of Radford et al., (2016) where images of
bedroom scenes generated by generative adversarial networks (GANs). Here there
is a single latent vector representation for the whole scene, which is not
disentangled across objects. This means, for example, that it is is very hard
to edit the latent representation to make specific changes in the scene (e.g.,
to change the colour of the bedspread), as the representation is not
interpretable.

To be clear, we are not arguing against the use of deep neural networks in
computer vision. However, for structured generative models DNNs can be used
for specific modelling and inference tasks (as we will see below), rather than
as one big black box. A similar point is made by Yuille and Liu, (2021), who
argue (their sec. 7.3) that to handle the combinatorial explosion of possible
images, computer vision systems need to be _compositional_ and _generative_ ,
and that this also leads to _interpretability_. These points are in excellent
agreement with the formulation of structured generative models.

|  |
---|---|---
input | latent variables | reconstruction
Figure 1: The input image (left) is explained in terms of 3D objects, the camera pose and illumination, to produce the reconstructed image (right). Images from Romaszko et al., (2020).  |  |  |
---|---|---|---
|  |  |
Figure 2: The left most panel shows two frames from a video of two people
walking past each other against a background. The second panel shows the mask
(top) and appearance of the first sprite learned. The third panel shows the
same thing for the second sprite. The rightmost panel shows the learned
background. Images from Williams and Titsias, (2004).

Fig. 1 shows how an input image can be explained in terms of 3D objects, the
camera pose and the illumination to produce the reconstructed image. Sometimes
a full 3D version may be too onerous, and we can consider a layered “2.1D”
model, where the objects (people) are represented by “sprite” models of the
shape and appearance, along with the background, as in Figure 2. The layers
have a depth ordering, so that occlusions can be explained.

The goals of this paper are: to promote the SGM viewpoint; to review relevant
work on object and scene modelling, and inference with SGMs; and to identify
gaps/outstanding issues where further research is needed. The structure of the
paper is as follows: In sec. 1.1 we discuss the rich variety of tasks
associated with scene understanding. Sec. 1.2 describes the general advantages
of generative models, and sec. 1.3 discusses the pros and cons of structured
generative models. Sec. 2 describes modelling objects, including “things”
(object categories that have a well defined shape) in sec. 2.1, including
parts-based models, and “stuff” (categories which have amorphous spatial
extent) in sec. 2.2. Sec. 3 covers models of the inter-relationships of
objects, focusing mostly on indoor scenes. Having defined models for objects
and scenes, sec. 4 discusses how inference for the SGM may be carried out,
from input consisting of a single or multiple images. Sec. 5 discusses issues
that are needed to advance the SGM agenda, including datasets and benchmarks.

Note that this paper covers a very large amount of ground, and does not aim to
provide comprehensive references for each topic. Indeed, the necessary topics
cover much of the content of a textbook on computer vision. Rather, it aims to
use prominent examples of work to provide an illustration of the various
topics.

|
---|---
(a) | (b)
Figure 3: Two example images 2008_001062 and 2008_000043 from the PASCAL VOC
2008 dataset.

###  1.1 Scene Understanding

Before discussing scene understanding more generally, let’s first look at two
example images in Fig. 3, and see what we can extract from them. These are not
particularly complex scenes—it would be easy to pick images with a lot more
objects and relations. Consider first the outdoor scene, Fig. 3(a)—we can
identify that this is not taken in a dense, “downtown” area, but equally not
in a very rural area. We can identify objects: a small herd of 6 cattle (one
is likely a calf mostly hidden behind the white-and-brown cow near the centre
of the image); 5 motor vehicles (one is half occluded); a building, some lamp
posts; some trees; and some road signs. This focus on objects may have
distracted us from the fact that large amounts of the images are “stuff”
categories, such as road surface and grass. The cows are on the grass, which
makes sense as they can graze there, but not on the road surface.

The second scene, Fig. 3(b), is an indoor scene of a dining room. We notice a
table, 6 chairs (of the same or similar design), 5 tablemats on the table, a
clock, a light fitting. The room has an outside door, a window, and a recessed
area and a large wooden cabinet (?) off to the left. There is a polished
wooden floor which allows some reflections, and the walls are a yellow-
greenish colour. Closer inspection would pick up smaller details like light
switches, a low-level radiator, some objects on the chairs (including perhaps
a child’s booster chair), and part of an indoor plant at the bottom right. We
can also see some outdoor railings through the window, suggesting that there
is a stairway up to the area outside the door.

So what is _scene understanding_? Part of it is about identifying the objects
and the stuff that are visible, but it is more than this. For each object we
would like to know its category, shape and its pose relative to the scene, and
the materials it is made of. We also would like to understand the camera
parameters (e.g., observer viewpoint) and lighting; this will help explain
occlusions and shadows. For example in Fig. 3(a) one can make inferences for
the direction of the sun, given the shadows of the cattle. Such a 3D
representation allows counterfactual questions, such as predicting how the
image would change if an object was removed or added, or if it was viewed from
a new direction (novel view synthesis, NVS). It also enables _interaction_ by
an agent in the scene, e.g., by attempting to herd the cattle.

Scene understanding also includes identifying the scene type (e.g., dining
room), which will give rise to expectations of what objects should (and should
not) be present. And it is about spatial, functional and semantic
relationships between objects.222See
https://ps.is.mpg.de/research_fields/semantic-scene-understanding. For example
in Fig. 3(a) we might find it surprising that the cattle are not fenced off
from the road to minimize collisions, but this may depend on the norms of the
location where the image was taken. And in Fig. 3(b) knowledge of dining rooms
means we would likely expect as many tablemats as chairs to be set on the
table—in fact close inspection of the image suggests that the “missing”
tablemat is has been placed on the seat of the chair on the right hand side.

One rich approach to scene understanding is via answering questions. In Visual
Question Answering (VQA; see, e.g., Antol et al., 2015), one typically expects
textual responses, but this might not be the best way to answer certain
questions, such as “which pixels belong to the black cow near the centre of
the image?”. The PASCAL VOC challenges (Everingham et al.,, 2010) asked three
questions: For _classification_ the question was “is there an object of class
X in the image?”. For _detection_ , the task was to predict the bounding box
of every object of class X in the image. And for _segmentation_ the task was
to label each pixel with one of the known class labels, or background. For the
last two, textual responses are not the most natural way to answer the
questions.

The epithet “a picture is worth a thousand words” also suggests that text is
not the most efficient way to describe a scene, particularly given the
ambiguities of natural language. Instead, SGMs provide a domain-specific
language for scenes. OpenAI’s system text-to-image system DALL-E (Ramesh et
al.,, 2021) can generate impressive output in response to prompts like “a
couple of people are sitting on a wood bench”, or even a quirky prompt like “a
tapir made of an accordion” (see Figs. 3 and 2(a) in the paper). However, it
has been reported that requesting more than three objects, negation, and
numbers may result in mistakes and object features may appear on the wrong
object (Marcus et al.,, 2022). An issue here is that in the paired text and
images sourced from the web, the text will likely not be sufficiently
informative about the objects and their spatial relationships etc. The kinds
of results produced by DALL-E and similar systems are thus unlikely to give
sufficient control to graphic artists and animators, who may be broadly happy
with the output of the system, but may wish to make adjustments and edits. In
order to enable this, we argue that one needs object-based representations and
scene models as advocated for above.

Figure 4: Image inpainting task, with the green rectangle blanked out, based
on image 2008_000959 from the PASCAL VOC 2008 dataset.

###  1.2 General Advantages of Generative Models

We first define some notation: $\mathbf{x}$ denotes observed data, such as an
image. A generative model $p_{\theta}(\mathbf{x})$ defines a probability
distribution over images; here $\theta$ denotes the parameters of the model.
For SGMs, our model is defined in terms of latent variables $\mathbf{z}$. The
latent variables for a single object might be decomposed, for example into
$\mathbf{z}=(\mathbf{z}^{s},\mathbf{z}^{t},\mathbf{z}^{p})$, for shape,
texture (including colour) and pose respectively. If there are $K$ objects,
they can each have latent variables (LVs)
$\mathbf{z}_{1},\ldots,\mathbf{z}_{K}$; let $\mathbf{z}_{0}$ denote the global
variables (e.g., camera parameters and illumination). There can also be
additional latent structure in $\mathbf{z}$ that models the inter-
relationships of objects in the scene. We have that

$p_{\theta}(\mathbf{x})=\int
p(\mathbf{z})p_{\theta}(\mathbf{x}|\mathbf{z})\;d\mathbf{z},$ (1)

where $p(\mathbf{z})$ is a prior over the latent variables, and
$p_{\theta}(\mathbf{x}|\mathbf{z})$ renders the latent description into the
image.333In eq. 1 the integral should be interpreted as a summation for latent
variables that are discrete.

There are several advantages of generative models, as discussed below.

##### Pattern synthesis (unconditional generation):

Generative models allow one to sample from the model, and compare the samples
to the raw input data. This can be very helpful, especially to identify ways
in which the model has failed to capture aspects of the input data, leading to
model revision. The methods of _model criticism_ (see e.g., Seth et al., 2018)
can help detect differences; one can also train a discriminator to do this, as
used in the training of GANs.

There are situations where such synthetic data (i.e. data sampled from the
model) can be useful in its own right—for example, in healthcare one may not
wish to release data due to privacy concerns. But if one can create synthetic
data which mimics the underlying data distribution, this can enable research
to proceed much more freely. However, one has to be careful that the synthetic
data has not simply “memorized” some or all of the training data, as discussed
in van den Burg and Williams, (2021).

##### Imputation and restoration (conditional generation):

If the input data $\mathbf{x}$ is split into observed data $\mathbf{x}^{o}$
and missing data $\mathbf{x}^{m}$, then the task of _imputation_ is to predict
the missing data given $\mathbf{x}^{o}$. Probabilistic models also produce a
probability distribution for $p(\mathbf{x}^{m}|\mathbf{x}^{o})$, which allows
a quantification of the uncertainty. In the case that part of an image is
missing, the imputation task can be called _inpainting_ ; see Fig. 4 for an
example. Here a simple method might just inpaint red and black texture from
the car body and tarmac, but knowledge of cars will predict a wheel in this
location, and it is likely that it will match in style to the visible front
wheel.

It might also happen that the observed data is a noisy or degraded version of
the underlying data; in this case having models of the underlying data and the
noise process allows probabilistic restoration of the data.

##### Anomaly detection:

It can be helpful to detect datapoints which do not conform to the learned
model $p(\mathbf{x})$. This task is known as _anomaly detection_ , _novelty
detection_ or _out-of-distribution (OOD) detection_. For example it can be
useful for an automated system to detect that the regime of operation has
changed, and thus flag up that it needs attention or re-training. One way to
frame this task is as a classification between $p(\mathbf{x})$ and a broad
(“crud-catcher”) model $p_{0}(\mathbf{x})$, as used e.g. in Quinn et al.,
(2009). If a data point $\mathbf{x}$ is more likely under $p_{0}(\mathbf{x})$,
it can be classified as an outlier relative to $p(\mathbf{x})$.

Anomalies can be quite subtle. In images of street scenes in North America
both fire hydrants and mailboxes are common items of street furniture, but it
is improbable to see a fire hydrant located on top of a mailbox—this example
of a contextual anomaly is from Biederman et al., (1982, Fig. 1).

##### Data compression:

A probabilistic model $p(\mathbf{x})$ can be used to compress data. Given the
true data distribution $p(\mathbf{x})$, Shannon’s source coding theorem will
assign a code of length $l(\mathbf{x})=-\log_{2}p(\mathbf{x})$ bits to
$\mathbf{x}$. Thus the expected code length is $-\int
p(\mathbf{x})\log_{2}p(\mathbf{x})\;d\mathbf{x}=H(p)$, the entropy of $p$.
Such data compression can be approached in practice using, for example, using
arithmetic coding, see e.g. MacKay, (2003, sec. 6.2).

In practice we may not know the true distribution $p(\mathbf{x})$, but have an
alternative model $q(\mathbf{x})$. In this case we have to pay a price in
terms of the expected number of bits used. Let the expected code length when
coding under $q(\mathbf{x})$ be denoted $L_{q}$. Then

$L_{q}=-\int p(\mathbf{x})\log_{2}q(\mathbf{x})\;d\mathbf{x}=-\int
p(\mathbf{x})\log_{2}\left(\frac{q(\mathbf{x})}{p(\mathbf{x})}p(\mathbf{x})\right)\;d\mathbf{x}=H(p)+D_{KL}(p||q)\geq
H(p).$ (2)

Hence the additional expected code length is given by the Kullback-Leibler
(KL) term, which of course reduces to zero when $q=p$. This motivates
minimizing the KL divergence to produce better codes, or equivalently to
maximize the expected log likelihood $\int p(\mathbf{x})\log
q(\mathbf{x})\;d\mathbf{x}$ for a model $q(\mathbf{x})$ (as the entropy term
is fixed).

###  1.3 Pros and Cons of Structured Generative Models

Below we contrast structured generative models (SGMs) compared to
discriminative models, or to unstructured generative models.

  * ✓

Structured generative models provide a coherent scene representation, rather
than just output predictions for a disparate set of tasks. This representation
is available for multiple tasks, including new ones not previously trained on
(transfer learning).

  * ✓

Structured generative models are _compositional_. This implies that when
learning about a particular object type, we don’t have to be concerned with
other object types at the same time. This should make them more data
efficient.

If there are inter-object relationships, these can be modelled separately from
the variability of individual objects, in a hierarchical model. This allows
curriculum learning (Bengio et al.,, 2009), where one can first focus on
modelling individual object classes using class-conditional data, and then
bring in within-scene inter-relationships. These advantages are not present in
an unstructured generative model.

  * ✓

The SGM representation is _editable_ , e.g., to change the direction of the
lighting, or to add/remove objects.

  * ✓

(Structured) generative models can be trained unsupervised, or with weak
supervision. Discriminative models usually require potentially
expensive/difficult human labelling, although the use weaker supervision has
also been explored (see, e.g., Shi et al., 2017).

  * ✓

The SGM is _interpretable/explainable_. This structured approach can be
contrasted with many deep generative models, which learn a rich model of the
data, but with a monolithic black-box architecture which is not interpretable
or easily editable. A SGM identifies certain image regions as being explained
by certain objects, and can potentially provide more detailed part-level
correspondences. The structured representation also enables other features
such as occlusion reasoning.

  * ✗

Discriminative models can be less susceptible to modelling limitations, as
they are directly optimizing for a given task “end-to-end”, rather than
building a general-purpose model which can be used for inference for many
different tasks.

  * ✗

The SGM framework can require expensive inference processes to infer the
latent variables for the whole scene. We discuss in section 4 below how these
issues can be ameliorated.

|
---|---
(a) | (b)
Figure 5: Images of an office scene from two viewpoints.

An example of where the SGM approach should be helpful is when an object is
heavily occluded, but scene context can help with its reconstruction. Consider
Fig. 5(a), where the rearmost red chair is heavily occluded.444This example
was inspired by Hueting et al., (2018). Knowledge that chairs grouped around a
table are often of the same design in such scenes would help make strong
predictions for this heavily occluded chair. This could be evaluated by
outputting a 3D model of the object, or making predictions of how the scene
would look from a novel viewpoint, as in Fig. 5(b). A related task is that of
image inpainting, as illustrated in Fig. 4. In this case there is in effect a
synthetic occluder (the mask); it is most natural to evaluate this by
prediction of the masked-out region(s).

The SGM approach could also be used to carry out scene editing, e.g. to add or
remove objects or change their properties, or alter the lighting. Here the
result could be evaluated by collecting views under the relevant perturbation.
Another possible task is the completion of a 3D scene given only a subset of
the objects in the scene (see e.g., Li et al., 2019). This is a missing data
imputation task, like image inpainting, but different as it is imputing a 3D
scene, not just an image.

##### Is full inference overkill?

The key advantage of the SGM approach is that it provides a unified
representation, from which many different tasks or questions can be addressed.
This creates a coherent understanding of the input image in terms of the 3D
(or 2.1 D) world, in contrast with what might arise if different models are
trained for different tasks without this underlying structure.

If we only care about one task, then it is certainly overkill. But acting in
the real world does not require just one task. The example of tea-making in
Land et al., (1999) illustrates this nicely. The goal of making tea decomposes
into subgoals such as “put the kettle on”, “make the tea”, “prepare the cups”.
A subgoal of putting the kettle on is to “fill the kettle”, and this in turn
requires “find the kettle”, “lift the kettle”, “remove the lid”, “transport to
sink”, “locate and turn on tap”, “move kettle to water stream”, “turn off tap
when full”, “replace lid”, and “transport to worktop”. The visual tasks
required include object detection; pose and shape estimation of objects so
that they can be manipulated; and monitoring of the state of some variable
(e.g. water level in the kettle). Carrying out the tea-making task in an
unfamiliar kitchen will bring in to play knowledge about the typical layout of
kitchens, and possibly about different kinds of tap mechanism. Note that
subgoals such as “locate and turn on tap” and “turn off the tap” are re-usable
across other tasks such as making coffee, or washing the dishes.

##  2 Models of Objects

Visual scenes can contain a lot of complexity. Components that make up the
scene can be divided into “things” and “stuff”. Things are object categories
that have a well defined shape (like people or cars), while stuff corresponds
to categories which have an amorphous spatial extent, such as grass and sky
(see e.g., Sun et al., 2014). We first focus on approaches to model things in
sec. 2.1, and then move on to model stuff in sec. 2.2.

###  2.1 Modelling Things: Multifactor Models

Here we consider modelling a class of visual objects (such as teapots or
cars). These can vary in shape, and in texture (described e.g. by its colour
or possibly a more complex pattern on the surface). We can also vary the
position and orientation (the pose) of the camera relative to the object, and
the lighting; we term these as _rendering_ variables. Hence there are separate
factors of _shape_ , _texture_ and _rendering_ that combine to produce the
observations. We call models with a number of separate factors multifactor
models. Below we first give some examples of multifactor models, and then
focus on parts-based models in sec. 2.1.1 (which are a special class of
multifactor models).

##### Example: Blanz and Vetter’s morphable model of faces.

An early example of a 3D multifactor model is due to Blanz and Vetter,
(1999).555The description below is partly based on Chapter 17 of Prince,
(2012) as well as the original paper. Consider $n$ locations on the face;
$\mathbf{s}_{i}$ records the $(x,y,z)$ coordinates of location $i$, and
similarly $\mathbf{t}_{i}$ records the red, green and blue colour values
(albedo) at the same location. These measurements were obtained with a laser
scanner. These individual vectors are concatenated to produce the shape vector
$\mathbf{s}=(\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{n})$ which has
length $3n$, and similarly there is an texture vector $\mathbf{t}$ of length
$3n$. Blanz and Vetter, (1999) used approximately 70,000 locations on each
face, and collected data from 200 subjects. Preprocessing was carried out to
remove the the global 3D transformation between the faces, and an optical flow
method was used to register the locations.

Given the shape and texture vectors for each subject, a probabilistic
principal components analysis (PPCA) model can be built to capture the
variation in shape and texture, with latent variables $\mathbf{z}^{s}$ and
$\mathbf{z}^{t}$ respectively. One could alternatively use a common
$\mathbf{z}$ for the shape and texture variation, e.g. by concatenating the
$\mathbf{s}$ and $\mathbf{t}$ vectors for each example before applying PPCA.
This could model the fact that a change in the shape of the mouth to produce a
smile will also likely expose the teeth to view, so these changes are
correlated.

The above description models shape and texture variation in 3D. This model is
transformed geometrically into the image plane in terms of the camera
intrinsic and extrinsic parameters. The colour at each pixel is determined by
the Phong shading model (see e.g., Szeliski, 2021, sec. 2.2.2), which accounts
for diffuse and specular reflections from directed light sources, and also for
ambient illumination. Denoting all of rendering variables by $\mathbf{z}^{r}$,
the overall model can be fitted to a new face by optimizing $\mathbf{z}^{s}$,
$\mathbf{z}^{t}$ and $\mathbf{z}^{r}$ so as to minimize an error measure
between the observed and predicted pixels.

The model of Blanz and Vetter, (1999) is in 3D. Such 3D models have also been
used, e.g., for modelling regions in the brain (Babalola et al.,, 2008). Some
earlier work by Cootes, Taylor and collaborators first developed 2D _active
shape models_ using a PPCA model of the shape as defined by landmarks (Cootes
et al.,, 1995), and then developed _active appearance models_ that also took
the texture into account (Cootes et al.,, 1998).

Figure 6: An illustration of the ability of CodeNeRF to carry out novel shape,
texture and pose synthesis. The 4 boxed images correspond to renderings with
shape and texture codes corresponding to the reference views. The other
results show renders obtained from the cross product of the shape and texture
codes, at a novel viewpoint. Image from Fig. 9 in Jang and Agapito, (2021)
licenced under CC BY 4.0.

##### Example: CodeNeRF models disentangled Neural Radiance Fields for object
categories.

Blanz and Vetter’s model uses a linear approach (PPCA) to model the
variability due to shape and texture. Careful alignment of the data was needed
in order to make this approach work. With the advent of deep learning, it is
natural to ask if one can exploit more powerful nonlinear models. We first
start with the Neural Radiance Field (NeRF) representation for a single object
due to Mildenhall et al., (2020), and then add latent variables to model shape
and texture variation, as in the CodeNeRF model of Jang and Agapito, (2021).

The NeRF takes as input a 3D location $\mathbf{x}=(x,y,z)$ and a viewing
direction defined by a 3D Cartesian unit vector $\mathbf{d}$, and outputs a
volume density $\sigma(\mathbf{x})$ and emitted colour
$\mathbf{c}(\mathbf{x},\mathbf{d})=(r,g,b)$ at that location and direction.
obtained via a neural network
$F_{\theta}:(\mathbf{x},\mathbf{d})\rightarrow(\mathbf{c},\sigma)$ with
weights $\theta$. As Mildenhall et al., (2020, sec. 4) state, “the volume
density $\sigma(\mathbf{x})$ can be interpreted as the differential
probability of a ray terminating at an infinitesimal particle at location
$\mathbf{x}$”. This means that the expected colour $C(\mathbf{r})$ observed at
the camera ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, with near and far
bounds $t_{n}$ and $t_{f}$ is given by666Note that in this section
$\mathbf{x}$ denotes a 3D location, not our usual meaning of an input image.
Also $t$ is overloaded to denote both the parameterization along the ray
$\mathbf{r}(t)$, and the texture superscript.

$C(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})\;dt,\qquad\mathrm{where}\qquad
T(t)=\exp\left(-\int_{t_{n}}^{t}\sigma(\mathbf{r}(s))\;ds\right).$ (3)

Here $T(t)$ denotes the transmittance along the ray from $t_{n}$ to $t$, i.e.,
the probability that the ray reaches $t$ starting from $t_{n}$ without hitting
any other particle. The process to obtain $C(\mathbf{r})$ is known as
_volumetric rendering_ , and is differtentiable. The computation of
$C(\mathbf{r})$ is approximated by taking a number of samples along the ray.

For a single object, the NeRF representation can be obtained by minimizing the
error between the observed and predicted colours at a set of ray locations for
a number of different views (with known camera poses and intrinsic
parameters). This is useful to allow novel view synthesis, i.e., to predict
the image that would be obtained from a novel view. However, for an object
class, it makes sense to have latent variables $\mathbf{z}^{s},\mathbf{z}^{t}$
for each object, as is done in CodeNeRF (Jang and Agapito,, 2021). Here the
neural network is enhanced to map
$F_{\theta}:(\mathbf{x},\mathbf{d},\mathbf{z}^{s},\mathbf{z}^{t})\rightarrow(\mathbf{c},\sigma)$.
Now the optimization problem for the network weights $\theta$ is carried out
over all training examples in an object class, and for each set of views of a
given example the shape and texture latent variables are estimated. (A
regularization penalty proportional to
$|\mathbf{z}^{s}|^{2}+|\mathbf{z}^{t}|^{2}$ is also imposed on the latent
variables, corresponding to a Gaussian prior.) The ability of CodeNeRF to
generalize to novel shape/texture/pose combinations is illustrated in Fig. 6.

There have been a lot of other recent developments arising from the NeRF work.
For example Zhang et al., (2021) extended NeRF to extract a surface (rather
than volumetric) representation, and then solve for spatially-varying
reflectance and environment lighting. This allows rendering of novel views of
the object under arbitrary environment lighting, and editing of the object’s
material properties.

##### Other multifactor models.

One can sometimes use a _multilinear model_ to handle multiple factors of
variation. A multilinear model with two latent factors $\mathbf{z}^{1}$ and
$\mathbf{z}^{2}$ is given by $x_{i}=\sum_{j,k}w_{ijk}z^{1}_{j}z^{2}_{k}$,
where $w_{ijk}$ is 3-way tensor of parameters. Tenenbaum and Freeman, (2000)
use a bilinear model to separate style and content, e.g. of the font (style)
of different letters (content). Vasilescu and Terzopoulos, (2002) use a
4-factor model to cover different facial geometries (people), expressions,
head poses, and lighting conditions.

Another example of a multifactor model is _transformation invariant
clustering_ (Frey and Jojic,, 2003). They consider the case where there is a
discrete factor modelling shape/appearance variation as a mixture model, and a
continuous factor arising from “nuisance” translations and rotations of an
object in an image.

####  2.1.1 Parts-based Models

Parts-based models are old idea in computer vision. For example Fischler and
Elschlager, (1973) described a model termed _pictoral structures_ , where an
object is represented by a number of parts arranged in a deformable
configuration. The deformations are represented by spring-like connections
between pairs of parts. Biederman, (1987) has advocated for a parts-based
approach in computer vision, under the name of _Recognition-by-Components_.
More recently Felzenszwalb et al., (2009) used discriminatively-trained parts-
based models to obtain state-of-the-art results (at the time) for object
recognition.

Some advantages of the parts-based approach are described by Ross and Zemel,
(2006), viz.

  * •

A partially-occluded object can be recognized if some of the parts are
present;

  * •

A parts-based approach is a good way to model the variability in highly-
articulated objects such as the human body;

  * •

Parts may vary less under a change of pose than the appearance of the whole
object.

  * •

Known parts may be recombined in novel ways to model a new object class.

Parts-based models share a similar objective with _perceptual organization_ or
_perceptual grouping_ (see, e.g., Palmer, 1999, ch. 6) in that they seek to
organize parts into a whole, but generally perceptual organization is seen as
a generic process, e.g., for grouping edges into contours, or to group similar
pixels into regions, rather than exploiting specific knowledge about certain
object classes.

Below we give four examples of parts-based models.

Figure 7: Top row: sample training images for Ross and Zemel’s models. Lower
rows: parts-based model learned by MCVQ. The plots on left show the masks, the
probability with which each pixel selects the given VQ. On the right are the
10 means for each VQ, multiplied by the mask shown on the left. Figures
reproduced from Ross and Zemel, (2006) with permission of R. S. Zemel.

##### Example: Parts-based Models of Faces.

One common application of parts-based models is to faces. In the UK, the
“PhotoFit” system is used by police in the investigation of crimes to produce
an image of a suspect with the help of eye witnesses. See, for example, the
“PhotoFit Me” work of Prof. Graham Pike.777See
https://www.open.edu/openlearn/PhotoFitMe.. This decomposes a face into eyes,
nose, mouth, jaw and hair parts.

Ross and Zemel, (2006) proposed two models to learn such a parts-based
decomposition. We describe them in relation to the modelling face
images.888The data used is aligned with respect to position and scale, so
these “nuisance factors” do not need to be modelled during learning. The first
model, Multiple Cause Vector Quantization (MCVQ), has $K$ multinomial factors,
each selecting the appearance of a given part. The “masks” for each part are
learned probabilistically, so that pixel $j$ is explained by part $k$ with
multinomial probability $\pi_{jk}$, where $\sum_{k}\pi_{jk}=1$. The learned
model is illustrated in Figure 7. The plots on the bottom left show the mask
probabilities, and on the bottom right the 10 means for each VQ are shown,
multiplied by the relevant mask. Notice in the bottom left panel how the masks
identify regions such as the eyes, nose and chin.

A second model is termed Multiple Cause Factor Analysis (MCFA)—this is similar
to MCVQ, but now the appearance each part is based on a factor analyzer
instead of a discrete choice. The mask model is as for MCVQ. Nazabal et al.,
(2022) used a MCFA model of faces, but added a higher-level factor analysis
model to correlate the factor analyzers for each part. The dataset used they
used (from PhotoFit Me) is balanced by gender (female/male) and by race
(Black/Asian/Caucasian), hence the high-level factor analyser can model
regularities across the parts, e.g. with respect to skin tone.

The MCVQ and MCFA models use a simple model for the mask probabilities
$\pi_{jk}$. But suppose we are modelling a parts decomposition of side-views
of cars, e.g. into wheels, body and windows. The different styles of cars will
give rise to quite different mask patterns, and these can be modelled with a
latent variable model. For example Eslami and Williams, (2011) modelled the
mask patterns with exponential family factor analysis, while Eslami and
Williams, (2012) used a multinomial Shape Boltzmann machine.

Figure 8: AND-OR graph for the clock object category. The dashed links between
the children of an AND node represent relations and constraints. Reproduced
from Zhu and Mumford, (2006, Fig. 6.1) with permission of now publishers inc.
©2006.

##### Example: Parts-based Model of a Clock.

Fig. 8 is reproduced from the work Zhu and Mumford, (2006). The figure
describes an AND-OR graph for clocks; for example there is an AND over the
hands, frame and numbers components of the clock, but in each there are
alternatives as encoded by OR nodes. Particular choices at the OR nodes gives
rise to a _parse graph_.999See sec. 3.3 for further discussion of grammars and
AND-OR graphs. One of the possible parse graphs is illustrated with dark
arrows, corresponding to the image at the top. Thus we observe that OR choices
are made for the hands (2 not 3), the numbers (Arabic not Roman), and the
shape of the frame (circular, and only the outer ring is present).

The AND-OR structure is similar to what we have seen for faces with the MCVQ;
there are a number of parts (the AND), and for each there are a discrete set
of choices (the OR). However, an AND-OR graph is generally more powerful, as
it can have hierarchical structure. For example, in the clock model there are
choices for all of the outer, inner, and central rings of the frame component
to be present or absent.

##### Example: Parts-based Models of Articulated Bodies.

A classic example of an articulated object is the human body, which can be
decomposed into a torso, head, arms and legs. The arms and legs can each be
further decomposed; for example an arm is made up of the upper arm, lower arm
and the hand, and the hand can be further decomposed into the fingers and
thumb. Given the importance of human avatars in the film and gaming
industries, there has been a lot of work on this topic. Here we focus on the
Skinned Multi-Person Linear Model (SMPL) of Loper et al., (2015). There are
two factors of variation that need to be taken into account. The first is the
pose of the body, as defined by the joint angles along the kinematic tree. The
second is the specific body shape of a given individual. The model for the
body is defined by a mesh with some 6890 vertices in 3D. Each vertex $i$ is
assigned weights $w_{ki}$ indicating how much part $k$ affects vertex $i$.

The body shape is modelled with a linear basis, similar to Blanz and Vetter’s
model discussed in sec. 2.1. The pose of the body is determined by the axis-
angle representation of the relative rotation of a part with respect to its
parent in the kinematic tree. One other important part of the model is _pose
blend shapes_ , which modify the the vertex locations depending on the pose,
but _before_ the pose transformation is applied. In SMPL, the pose blend
shapes are a linear function of the elements of the part rotation matrices.
Pose blend shapes are needed to counter the unrealistic deformations at joints
that would otherwise arise. After combining the shape and pose blend effects
in the neutral pose, the final predicted location of vertex $i$ is obtained as
a weighted sum (with weights $w_{ki}$) of the transformation of vertex $i$
under part $k$.

##### Example: Capsules

Another parts-based model is termed “capsules”. This term was introduced in
Hinton et al., (2011), with later developments including Sabour et al.,
(2017), Hinton et al., (2018) and Kosiorek et al., (2019). There is a recent
survey paper on capsules by De Sousa Ribeiro et al., (2022). The term
“capsule” relates to a visual entity which outputs both the probability that
the entity is present, and a set of instantiation parameters for the entity.
Although capsules are usually described in an inferential manner, with the
flow of information from the parts to the object, below we follow the
exposition of Nazabal et al., (2022) who described Generative Capsule Models.

Consider an object template $T$ which consists of $N$ parts
$\\{\mathbf{p}_{n}\\}_{n=1}^{N}$. Each part $\mathbf{p}_{n}$ is described by
its class, pose, shape, texture. The template $T$ has an associated latent
variable vector $\mathbf{z}$ which affects to the pose, shape, and texture of
the parts. For example for a template in 2D, part of $\mathbf{z}$ may define
the parameters of a similarity transformation (in terms of a translation
$\mathbf{t}$, rotation $\theta$ and scaling $s$ of the template). The
geometric transformation between the object and the parts can be described by
a linear transformation in terms of $\mathbf{t}$, $s\cos\theta$ and
$s\sin\theta$. Other parts of $\mathbf{z}$ can model shape and texture
correlations between parts. Methods for matching observed parts to template
parts are described in sec. 4.

Although we have described here object-parts relationships, capsules can be
formed into an hierarchical architecture, allowing e.g., the representation of
object inter-relationships.

###  2.2 Modelling Stuff: Visual Texture

Forsyth and Ponce, (2003, p. 164) discuss texture as follows:

> Texture is a phenomenon that is widespread, easy to recognise, and hard to
> define. Typically, whether an effect is referred to as texture or not
> depends on the scale at which it is viewed. A leaf that occupies most of an
> image is an object, but the foliage of a tree is a texture. Views of large
> numbers of small objects are often best thought of as textures. Examples
> include grass, foliage, brush, pebbles, and hair. Many surfaces are marked
> with orderly patterns that look like large numbers of small objects.
> Examples include the spots of animals such as leopards or cheetahs; the
> stripes of animals such as tigers or zebras; the patterns on bark, wood, and
> skin. Textures tend to show repetition: (roughly!) the same local patch
> appears again and again, though it may be distorted by a viewing
> transformation.

Textures can be classed as regular or stochastic, although there can be
gradations, such as near-regular or near-stochastic. Examples of regular
textures include brickwork, tiled floor patterns, and wickerwork. Examples of
stochastic textures include clouds, wood grain, and foliage. Below we will
focus mainly on stochastic textures.

We take the goal of _texture synthesis_ to be the generation of an arbitrarily
sized region of a texture, given a (small) training sample. For regular
textures it should be possible to extract the repeating element(s) and simply
tile the target region appropriately, but this will not work for stochastic
textures. Instead we aim to learn a generative model of the texture from the
training sample. A common type of model used is an _energy-based model_ (EBM).
Let $\mathbf{x}_{(k)}$ denote a patch of the image centered at location $k$
(e.g. a square patch). The _field of experts_ (FoE) energy is defined as

$E_{FoE}(\mathbf{x})=\sum_{k}\sum_{j}\phi_{j}(\mathbf{w}_{j}\cdot\mathbf{x}_{(k)}),$
(4)

where $\mathbf{w}_{j}$ is a filter the same size as the patch, and
$\phi_{j}()$ is some function. A probability distribution over images is then
defined by the Boltzmann distribution, i.e.,

$p_{FoE}(\mathbf{x})=\frac{1}{Z(W)}\exp(-E_{FoE}(\mathbf{x})).$ (5)

Here $Z(W)$ is the _partition function_ that serves to normalize
$p_{FoE}(\mathbf{x})$. The term field of experts was introduced in Roth and
Black, (2005), as a generalization of the product of experts construction due
to Hinton, (2002), to handle arbitrarily-sized images.

In general it is not easy to draw samples directly from an energy-based model;
a standard approach is to construct a Markov chain whose equilibrium
distribution is the desired Boltzmann distribution. See e.g., Murphy, (2012,
ch. 24) for further details.

One simple choice would be to take the function $\phi_{j}$ to be a quadratic
form in $\mathbf{x}_{(k)}$. If this is positive definite,
$p_{FoE}(\mathbf{x})$ will be well-defined and normalizable, and will define a
Gaussian Markov random field (GMRF), see e.g., Rasmussen and Williams, (2006,
sec. B.5). Stationary GMRFs on a regular grid can be analyzed via Fourier
analysis (Rozanov,, 1977) in terms of the power spectrum. However, it is well
known that image models based on simply on the power spectrum (or
equivalently, on second-order statistics, via the Wiener-Khinchine theorem)
are inadequate. For example, Fig. 2 in Galerne et al., (2011) shows a section
of a tiled roof. Randomizing the phase of its Fourier transform, while
maintaining the power spectrum, leads to a blurry image, as the phase
alignment needed to create sharp edges no longer occurs.

A Gaussian random field model can also be obtained via the _maximum entropy
principle_ (see e.g., Cover and Thomas, 1991, ch. 12) on the basis of second
order (covariance) constraints on the distribution. But an alternative is to
consider a set of filters, and impose the constraint that the maximum entropy
model matches the observed histogram for each filter. This gives rise the
FRAME (Filters, Random field, and Maximum Entropy) model of Zhu et al.,
(1998). Roth and Black, (2005) discuss the FRAME model, and note that the
approach is complicated by its use of discrete filter histograms. Instead they
propose the field of experts model.

Raw |  |  |  |  |  |
---|---|---|---|---|---|---
Tm |  |  |  |  |  |

D4 D68 D103

Figure 9: The top row shows two sample patches from each of three textures
(labelled D4, D68 and D103). The bottom row shows two samples drawn from a Tm
model trained on the specific textures. Figure from Kivinen and Williams,
(2012, Fig. 4).

Kivinen and Williams, (2012) defined a different energy based model, with the
energy function

$E_{Tm}(\mathbf{x})=\frac{1}{2\sigma^{2}}(\mathbf{x}-\mathbf{a})^{T}(\mathbf{x}-\mathbf{a})-\sum_{k,j}\log[1+\exp(b_{j}+\sigma^{-1}(\mathbf{w}_{j}\cdot\mathbf{x}_{(k)})].$
(6)

This was inspired by the convolutional restricted Boltzmann machine of Lee et
al., (2009), but instead of convolution uses _tiled convolution_ as described
in Ranzato et al., (2010), who argue that convolutional weight sharing creates
problems due to nearby latent variables of the same filter being highly
correlated. In the tiled convolutional strategy each filter tiles the image
with copies of itself without overlaps (i.e. the stride equals the filter
diameter). But different filters do overlap with each other, in order to avoid
tiling artifacts. The energy function $E_{Tm}$ consists of two terms. The
first component corresponds to a simple spherical Gaussian with mean
$\mathbf{a}$. The second is obtained by integrating out the hidden units (as
given in eq. 1 of Kivinen and Williams, 2012) of the restricted Boltzmann
machine analytically. The label “Tm” is given to this model in order to denote
that it uses tiled convolution, and that the means, rather than the
covariances, are modelled (by the $\\{\mathbf{w}_{j}\\}$s and $\mathbf{a}$).

Kivinen and Williams, (2012) showed that to model multiple textures one can
keep a fixed set of filters, but adjust the biases on a per-texture basis—they
called this the “Multi-Tm” model. Fig. 9 shows results for three different
textures, with two images of both the raw data and samples. The raw data is
obtained as $98\times 98$ patches cropped from the Brodatz texture album
(Brodatz,, 1966).

An alternative to energy based models is an auto-regressive model, where the
predicted pixel value $x_{ij}$ at location $(i,j)$ depends on a vector of
context, typically to the left and above if working in raster scan order.
Efros and Leung, (1999) used this approach to carry out _exemplar-based
texture synthesis_ , starting with a source training sample of texture. For a
target location $(i,j)$, one identifies neighbourhoods in the source texture
that are similar to the current context region, and then selects one of these
regions at random (depending on the level of agreement with the context
region). For filling in holes an “onion peeling” strategy of scanning round
the periphery can be used rather than raster scan order. Note that this is a
non-parametric modelling approach—rather than constructing a parameterized
model for $p(x_{ij}|\mathrm{context})$, one extracts relevant regions from the
source texture.

Auto-regressive models do not have to be exemplar-based. PixelCNN (van den
Oord et al.,, 2016) and PixelCNN+ (Salimans et al.,, 2017) are prominent
recent auto-regressive models which use convolutional layers for feature
extraction, along with masking to ensure that only valid context regions are
accessed, in order predict $p(x_{ij}|\mathrm{context})$.

Above we have discussed the generation of flat textures on a 2D plane.
Textures can be applied (mapped) to a surface; this is known as _texture
mapping_ (see, e.g., Szeliski, 2021).

##  3 Models of Scenes

A very simple model of scenes is one which randomly selects objects and puts
them into a scene. This might be summarized as “the independent components of
images are objects”. One example of this is the 2D sprites work of Williams
and Titsias, (2004) illustrated in Fig. 2. Here the model has learned about
the background and the two people, but a priori it would place the people at
random locations in the image. A more recent example is IODINE (short for
Iterative Object Decomposition Inference NEtwork) due to Greff et al., (2019).
This uses an “object-centric” representation, consisting of $K$ vectors of
latent variables $\mathbf{z}_{1},\ldots,\mathbf{z}_{K}$, one for each object.
IODINE was demonstrated on 2D sprites data, and also on images of 3D scenes
from the CLEVR dataset, which consists of geometric objects like spheres and
cubes in random locations with random material properties. The
$\mathbf{z}_{k}$s for each object in IODINE were not factored into shape,
texture and pose components (as discussed in sec. 2.1), but were a single
vector that entangled these factors.

However, in the same way that sentences are not random sequences of
words,101010are sequences sentences of not words random! visual scenes are not
composed of random collections of objects—there are co-occurrences of objects,
and relationships between them. For example, there are correlations between
the scene type (e.g. kitchen. living room, urban street, rural field) and the
kinds of objects observed. Also, there are stuff-stuff, things-stuff, and
thing-thing interactions that occur between objects in the scene (see e.g.
Heitz and Koller, 2008). Examples of things-stuff interactions are that cars
are (usually) found on roads, or cows on grass. An example of thing-thing
interactions is that dining chairs are likely to be grouped around a dining
table. As another example of thing-thing interactions, one might consider
adding details to a coarse scene layout, e.g. by adding tablemats, cutlery and
crockery to the dining table.111111 Ritchie, (2019) terms this virtual “set
dressing”. The reader will observe that there are similarities between parts-
based models described in sec. 2.1.1, and scene models. However, parts-based
models are often more constrained, with a fixed number of parts, while scene
models can have a variable number of objects and looser relationships. Scene
relationships can also be longer-range—a classic example is the relationship
between a TV and a sofa for viewing it, which need to be a comfortable
distance apart.

Below we focus particularly on models for indoor scenes, reflecting the focus
in the research literature. But there are also outdoor scenes, in both rural
and urban environments. In scenes of mountains or coastlines, it may be most
natural to consider the carving of the landscape by erosion, e.g. by river
valleys. In farmland there will be human-made field boundaries (constrained by
the landscape), along with crops or livestock. In urban environments, one
might use grammar-type models to generate building facades, and then
“decorate” the street architecture with other objects such as people, cars and
street furniture.

Below we describe autoregressive, energy based and hierarchical models, which
we cover in turn. Note that autoregressive and energy based models are not
latent variable models, while hierarchical models are. In this section it is
assumed that 3D data is available, e.g. 3D oriented bounding boxes (OBBs) with
class labels.

###  3.1 Autoregressive Models

We take as an example the work of Ritchie et al., (2019), who describe a
process where objects are added to a room layout one at a time, until a
decision is made to stop. The model first extracts a top-view floor plan of
the room (to define the valid region to place objects). It then feeds the
floor plan to a sequence of four modules that (i) decide which object (if any)
to add, (ii) specify where the object should be located, (iii) its
orientation, and (iv) its physical dimensions. Once an object has been added
the floor plan representation is updated to include the object, before the
next calls to steps (i) to (iv). These modules are implemented with
convolutional neural networks.

Let the $m$ ordered objects be denoted
$\mathbf{x}_{1},\;\mathbf{x}_{2},\ldots,\mathbf{x}_{m}$. Each $\mathbf{x}_{i}$
is comprised of an object class label, pose features (location and
orientation), shape features and texture features121212Ritchie et al., (2019)
do not use texture features, but in general they could be present., so that
$\mathbf{x}_{i}=(\mathbf{x}_{i}^{c},\mathbf{x}_{i}^{p},\mathbf{x}_{i}^{s},\mathbf{x}_{i}^{t})$.
Then under an autoregressive model we have that

$p(\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{m})=p(\mathbf{x}_{1})\prod_{i=2}^{m}p(\mathbf{x}_{i}|\mathbf{x}_{<i}),$
(7)

where $\mathbf{x}_{<i}$ denotes the sequence
$\mathbf{x}_{1},\ldots,\mathbf{x}_{i-1}$. This is suitable for generating
scenes from the model. It can also be used to compute
$p(\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{m})$ if an ordering of the
objects is given. However, when we observe an _unordered_ set of objects
$X=\\{\mathbf{x}_{i}\\}_{i=1}^{m}$, we should sum over all possible
permutations, so that

$p(X)=\sum_{\pi\in\Pi}p(\pi)p(\mathbf{x}_{\pi(1)})\prod_{i=2}^{m}p(\mathbf{x}_{\pi(i)}|\mathbf{x}_{\pi(<i)}),$
(8)

where $\pi$ denotes a particular permutation,and $\Pi$ is the set of all
permutations. The prior over permutations $p(\pi)$ can be taken as uniform,
i.e. $1/(m!)$. In fact Ritchie et al., (2019) do use an ordering of objects
for training the object class label module, based on a measure of the
importance of the class. This depends on the average size of a category
multiplied by its frequency of occurrence. This means that large objects like
a bed will occur first in a bedroom scene, with other objects fitting in
around it.

The model of Ritchie et al., (2019) is relatively simple, but the use of the
floor plan representation with the added objects means that the chain rule of
probabilities can be used readily to model the context, without
simplifications such as a Markov model.131313Although note that the floor plan
representation discards the order in which objects were added. One can ask if
the autoregressive process really makes sense as a generative model of scenes?
It is not unreasonable that large objects (such as the bed in a bedroom scene)
should be added first. But it seems rather unlikely that people furnish
bedrooms using a fixed ordering of all the object classes.

Note also the autoregressive model does not make explicit groupings of objects
that co-occur, such as a bed and nightstand, or a tv-sofa combination arranged
for convenient viewing. In contrast, hierarchical models (see below) should be
able to pick out such structure.

Building on the work of Ritchie et al., (2019), Paschalidou et al., (2021)
develop an autoregressive model using transformers (dubbed ATISS). They
demonstrate some advantages of ATISS over the earlier work, e.g. in relation
to scene completion, especially when objects that come early in the sequence
(e.g., beds for a bedroom) are omitted. However, in the formulation of ATISS
(their eq. 3), a valid sum over permutations of the sequence as in eq. 8 is
replaced by a product over permutations. The motivation is to maximize the
probability of generation over all possible orderings, rather than having at
least one with high probability. But taking a product over sequences is not a
valid probability calculation; one possible way to make sense of this is as a
product-of-experts (PoE) construction (Hinton,, 2002), although in this case a
partition function $Z$ should be introduced, which would complicate the
learning.

###  3.2 Energy-based Models

Suppose that we have generated a number of objects to go in a room. We then
need to arrange them in order to obey a number of spatial, functional and
semantic relationships or constraints. The basic idea is to define an energy
function that measures the fit of the configuration to these constraints, and
then as per eq. 5 to define a probability distribution using the Boltzmann
distribution. Yu et al., (2011) used such an approach to automatically
optimize furniture arrangements, using simulated annealing to search in the
configuration space.

The method of Yu et al., (2011) is defined for a fixed set of objects. Yeh et
al., (2012) extended this idea to allow an “open world”, where the number of
objects is variable. They used a reversible jump Markov chain Monte carlo
(MCMC) method from this space.

A problem with energy based models in general is that one generally needs to
run a MCMC chain for many iterations in order to draw samples from the
equilibrium distribution. This limitation applies to the above methods,
meaning EBMs will not scale well to larger scenes.

###  3.3 Hierarchical Models

A natural way to obtain an hierarchical model is via a grammar-based approach.
The idea of using grammars for pattern analysis is an old one, see for example
the work of K. S. Fu (1982) on syntactic pattern recognition. In relation to
scene understanding, perhaps the most notable work is from Song-Chun Zhu and
collaborators. The long paper by Zhu and Mumford, (2006) entitled “A
Stochastic Grammar of Images” is a key reference. A parts-based model of
clocks due to Zhu and Mumford, (2006) is discussed above in sec. 2.1.1.

Figure 10: A hierarchical grammar for bedroom scenes, redrawn from Liu et al.,
(2014).

A context-free grammar (CFG) is defined in terms of terminal symbols, non-
terminal symbols, a start symbol, and production rules. Inspired by Liu et
al., (2014), we take bedroom scenes as an example domain. Here we will have
terminal symbols for observed objects such as mattress, nightstand, pillow
etc. As above, the full description will involve not only the class label, but
also pose, shape and texture information. Non-terminal symbols here would
correspond to groupings of objects; for example we may have
$\mathrm{sleeping\mathchar 45\relax
area}\rightarrow\mathrm{bed}\;\mathrm{nightstand\mathchar 45\relax group}$.
There may typically be one or two nightstands (usually depending on whether
the bed is single or double), so that the productions
$\mathrm{nightstand\mathchar 45\relax group}\rightarrow\mathrm{nightstand}$
and $\mathrm{nightstand\mathchar 45\relax
group}\rightarrow\mathrm{nightstand}\;\mathrm{nightstand}$ are both valid. An
example grammar for bedroom scenes from Liu et al., (2014) is shown in Figure
10.

A probabilistic context-free grammar (PCFG) adds a probability distribution
over the productions which share the same non-terminal on the LHS of the
rules. Charniak, (1993, ch. 5) provides a good overview of CFGs and PCFGs. A
CFG is often described in terms of a set of production rules, but it can also
be described by an AND-OR graph; Hall, (1973) showed the equivalence between
the two. Here, the OR occurs over productions share the same non-terminal on
the LHS; the AND occurs with productions with more than one symbol on the RHS,
as all of these symbols must be generated. As Zhu and Mumford, (2006, sec.
6.1) state, an “AND-OR graph embeds the whole image grammar and contains all
the valid parse graphs”.

We have seen that the terminal symbols include class label, pose, shape and
texture information in general. This means that the non-terminals which govern
a set of terminals will need to include latent variables which can generate
the appropriate correlations between them. For example the production
$\mathrm{sleeping\mathchar 45\relax
area}\rightarrow\mathrm{bed}\;\mathrm{nightstand\mathchar 45\relax group}$
will need to specify the size of the bed (single or double), and this
information will also need to be passed to the $\mathrm{nightstand\mathchar
45\relax group}$ variable, so that it can determine whether to generate one or
two nightstands (and their appropriate location(s)). In the CFG, pose and size
information can be defined relative to higher level variables. So, for
example, if the size and location of the $\mathrm{sleeping\mathchar 45\relax
area}$ has been defined, it makes sense to locate the bed relative to this. A
possible alternative to encoding this information in the latent variables is
to add horizontal relational structures that to encode contextual information
or constraints between nodes, as proposed in Zhu and Mumford, (2006, ch. 4)
and Jin and Geman, (2006).

A problem with using CFGs to model visual scenes is that it can be hard to
learn them from data. Even in natural language processing (NLP), most
successful grammar learning uses annotated “treebank” data, rather than
unannotated sentences. Similarly, annotated data has been used to learn some
hierarchical models for images, as in Yao et al., (2007) and Liu et al.,
(2014). But recently for NLP data, Kim et al., (2019) found that they could
add a sentence-level continuous vector latent variable in addition to the PCFG
structure (to produce a “compound” PCFG) yielding state-of-the-art results for
parsing tasks.

While grammars are one way to generate hierarchical structure, they are not
the only way. Instead of discrete-valued non-terminals, one can use a
continuous-valued latent vector in a node. Consider starting with a single
latent vector $\mathbf{z}$. The first binary split can be generated as

$[\mathbf{z}_{1},\mathbf{z}_{2}]=\mathbf{f}(\mathbf{z}),$ (9)

where $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ are the latent vectors of the two
children., and $\mathbf{f}$ is a non-linear vector-valued function, which
could be as simple as
$[\tanh(W_{1}\mathbf{z}+\mathbf{b}_{1}),\tanh(W_{2}\mathbf{z}+\mathbf{b}_{2})]$
or a deeper neural network. For each generated latent vector, a binary node
classifier is applied to decide whether it is a terminal (generating an
object), or a non-terminal which can be further split. The advantage of the
continuous state is that it is well suited to express pose, shape and texture
variation. This kind of generative structure is a simplified version of the
GRAINS model of Li et al., (2019) which is described below. Note that the
hierarchical MCFA model for faces of Nazabal et al., (2022) (described in sec.
2.1.1) is a simple one-layer model of this type, but with $K$ child nodes
rather than just two, and without the $\tanh$ nonlinearities.

Li et al., (2019) developed the Generative Recursive Autoencoders for INdoor
Scenes (GRAINS) model. This takes as input an unstructured set of objects in
3D and computes a latent representation $\mathbf{z}$, which is decoded to
yield a hierarchical structure of objects which can then be rendered. In more
detail, the GRAINS model decodes tbe latent vector $\mathbf{z}$ into five
variables representing the floor and four walls of a room. Each of these is
either a terminal (corresponding to an object), or a non-terminal, as
determined by a node classifier neural network. There are number of non-
terminals, corresponding to support, surround and co-occurrence relationships.
If the model were a grammar the encoder network should sum over all parses of
the data, but instead a tree structure is built heuristically bottom up, and
that is used in the encoder to output $\mathbf{z}$.

##  4 Inference

In this section the task is inference from a single RGB or RGBD image, or from
a set of images with known viewpoints (multi-view data), assuming that the
object and relational models have already been learned. Thus we are interested
in $p(\mathbf{z}|\mathbf{x})$.

We first cover inference for objects and global variables, and then inference
with scene models. Finally we discuss inference in the presence of model
deficiency, where an input image cannot be reconstructed exactly.

##### Inference for Objects and Global Variables.

The basic inference task is to (i) detect each object, and to determine its
class label, shape, texture and pose information, (ii) detect and characterize
regions of stuff, and (iii) determine the lighting and camera global latent
variables. The inference problem for a single object is already daunting if we
we consider a discretization of the values for each of the latent variables
and search over these (see e.g. Yuille and Liu, 2021 sec. 7.1), and there is a
combinatorial explosion when considering multiple objects and the global
variables. The most direct way to address the combinatorial explosion is to
_approximate_ the search. One such method is Markov chain Monte Carlo (MCMC),
where the goal is construct a Markov chain whose equilibrium distribution
samples from the desired posterior over the latent variables. See e.g.,
Murphy, (2012, chapter 24) for a discussion of MCMC. Often the proposed moves
in the state-space are generic and do not exploit the structure of the
problem, leading to slow mixing. An example of a more advanced approach is
Data-driven Markov chain Monte Carlo (DDMCMC, Tu and Zhu, 2002), which allows
proposals such as the candidate set of object detections to be incorporated
into a valid MCMC algorithm.

An alternative approximate approach is to use _variational inference_ , see
e.g. Jordan et al., (1999), where the goal is approximate the posterior
$p(\mathbf{z}|\mathbf{x})$ with a variational distribution $q(\mathbf{z})$.
The variational autoencoder (Kingma and Welling,, 2014; Rezende et al.,, 2014)
uses _amortized_ variational inference to predict a distribution over the
latent variables given input data. This is achieved with an “encoder” network
(a.k.a. a “recognition model”, see Dayan et al., 1995). Such amortized
inference is relatively straightforward if there is one object of interest in
the image, but is more complex if there are multiple objects, due e.g. to
permutation symmetries. In IODINE (Greff et al.,, 2019) the feed-forward
predictions from the image for the object latent variables are the refined
iteratively to take into account effects such as explaining away. In the slot-
attention model of Locatello et al., (2020), an iterative attention mechanism
is used in the mapping from the inputs to the latent variables, so that they
_compete_ to explain the objects in the image. The attend, infer, repeat (AIR)
model of Eslami et al., (2016) takes an alternative, sequential approach,
identifying one object at a time.

MCMC and variational inference (VI) methods can be used to express the
uncertainty in the latent representation $\mathbf{z}$ given the data
$\mathbf{x}$. This may arise, e.g. due to (partial) occlusions, and can give
rise to a multi-modal posterior, corresponding to different interpretations or
“parses” of the input image. A limitation of VI methods is that they may not
capture this multi-modality well if the assumed form of $q(\mathbf{z})$ is
unimodal. As explained in Murphy, (2012, sec. 21.2.2), when the variational
distribution is unimodal, then the Kullback-Leibler divergence $KL(q||p)$ used
in VI tends to pick out one mode of the posterior, and thus under-estimate the
uncertainty. MCMC methods can in principle sample from a multi-modal
posterior, but in practice can get trapped within one mode, unless special
measures such as annealing (see e.g., Murphy, 2012, sec. 24.6) are used. One
way to reduce the uncertainty in $q(\mathbf{z})$ is to fuse information from
multiple views, as shown in Li et al., (2020).

The above models such as IODINE and AIR were trained unsupervised. But the
development of object detectors over the last two decades means that one can
train object detectors for known classes such as cars, pineapples etc. The
output may be a bounding box (BBox), or possibly a region-of-interest. Such an
approach was used by Izadinia et al., (2017) in their IM2CAD system, which
takes as input an image of a room. It estimates the room geometry (as a 3D
box), detects objects, predicts the associated latent variables of each
object, places the objects in the scene, and then optimizes their placement
via rendering the scene and comparing it to the input image.

If the input data is a set of multi-view images, then geometric computer
vision techniques can be brought to bear. Simultaneous localization and
mapping (SLAM) or structure-from-motion (SfM) can be used to estimate camera
poses and a sparse point cloud representation, as in COLMAP (Schönberger and
Frahm,, 2016). By themselves, such techniques do not segment the data into
objects, but they can be combined with object-detection bounding boxes and
masks predicted from each image to produce a 3D bounding box, as in Rünz et
al., (2020, sec. 5.1). Once the objects are segmented, their shape, texture
and pose latent variables can be estimated, see e.g., Rünz et al., (2020,
secs. 5.2-5.3).

One important recent development has been _differentiable rendering_ , which
allows optimization of the estimated latent state, in order to improve the fit
between rendered scene and the input image(s). An early differentiable
renderer was _OpenDR_ due to Loper and Black, (2014). Kato et al., (2020)
provide a survey of various methods that have been proposed for mesh, voxel,
point cloud and implicit surface representations.

##### Inference with Scene Models.

Above we have discussed instantiating objects and the camera/lighting global
variables; we now consider inference for _relational_ structures. For example,
these may be represented as a parse graph that needs to be inferred on the
fly.

One of the major issues is to match a set of object and part detections to a
scene structure like an AND-OR graph. As a simple example, consider a capsules
model for an object, as discussed in sec. 2.1.1. This is comprised of a number
of parts that lie in a certain geometric relationship to each other. (Similar
arguments also apply to a grouping a objects in a scene, such as a computer-
monitor-keyboard, but here we will use the terminology of an object and its
parts.)

Now consider that we have detected a set of parts
$\\{\mathbf{x}_{m}\\}_{m=1}^{M}$, and the task is to match these to a set of
templates $\\{T_{k}\\}_{k=1}^{K}$. Let $w_{mnk}\in\\{0,1\\}$ indicate whether
observation $\mathbf{x}_{m}$ is matched to part $n$ of template $k$. The $w$’s
can be considered as a binary matrix $W$ indexed by $m$ and $n,k$. One way to
frame this assignment problem is to consider valid _matchings_ between
observed and template parts, as defined by a permutation matrix. If $M$ is not
equal to the total number of model parts $N=\sum_{k=1}^{K}N_{k}$, then dummy
rows or columns can be added to make the problem square, e.g., in case some
parts are missing. As the exact computation would require considering all
possible permutations which scales exponentially with $M$, Nazabal et al.,
(2022) consider variational inference for the match variables $W$ and the
latent variables $\mathbf{z}_{k}$ for each template. This implements a
_routing-by-agreement_ procedure, as discussed by Sabour et al., (2017) and
Hinton et al., (2018), but derived from the variational inference equations
rather than as an ad hoc objective. An alternative to routing-by-agreement is
to use a random sample consensus approach (RANSAC, Fischler and Bolles, 1981),
where a minimal number of parts are used in order to instantiate an object
template, which is then verified by finding the remaining parts in the
predicted locations. Kosiorek et al., (2019) use another inference approach,
where an autoencoder architecture predicts the LVs for each template, making
use of a Set Transformer architecture (Lee et al.,, 2019) to handle the
arbitrary number of observations $M$.

If the hierarchical structure has been defined in terms of a recursive neural
network, as in GRAINS (Li et al.,, 2019), then it is possible to build an
encoder to predict the scene latent variable $\mathbf{z}$, which can then be
decoded to produce the hierarchical structure. This was successful in GRAINS,
but note that there the input is a set of segmented 3D objects, as opposed to
an unsegmented image.

Although it is natural to think of inferential information flows in the
hierarchical model being bottom-up, from parts to objects to scenes, it does
not have to happen this way. The overall scene type may be well-characterized
by a global scene descriptor like the _gist_ (Oliva and Torralba,, 2006), and
this will create, e.g., top-down expectations for certain object classes, and
not others. For example, Torralba et al., (2010) made use of the gist to
predict the scene type (such as beach scene, street scene, living room). This
then made useful predictions for the vertical location in the image for
objects of a certain class (e.g. cars in street scenes)141414In the scene
types considered, the horizontal location of objects was usually not well
constrained by the scene type.. This indicates more generally that information
from the image(s) may flow in top-down as well as bottom-up fashion for
inference in the scene model.

As Zhu and Huang, (2021, sec. 1.3.4) note, vision is driven by a large number
of tasks, and “Each of these tasks utilizes only a small portion of the parse
graph, which is constructed in real-time in a task-driven way based on a
hierarchy of relevant tasks”. Thus it may be useful to consider “lazy
inference”151515Thanks for Kevin Murphy for suggesting this term. of the full
parse graph.

##### Model deficiency:

Quoting George Box, we can say that “all models are wrong, but some are
useful” (Box and Draper,, 1987, p. 424). There is a great richness and detail
in many visual scenes, and it may not be possible (or perhaps even desirable)
to model this fully. However, if we cannot get zero error at the pixels, to
what extent can a generative model be said to have fully explained the image?
The issue here is that there can be many ways in which an aggregate measure of
error (such as mean squared error, MSE) can arise: (i) the pose of an object
could be slightly off, but the appearance and shape are correct, leading to a
“halo” of errors around the boundaries of the object; (ii) the pose and shape
may be correct, but the object’s texture does not match the palette of known
textures, and thus there is a high-frequency pattern of errors within the
object’s extent; (iii) an object’s shape is incorrect (either due to failures
in inference or modelling), but the pose and texture are correct; or (iv)
there may have been a false positive or false negative detection of a small
object, again leading to the same MSE.

A natural way to tackle this problem is to compare the input and predicted
images, along with the predictions for object extent etc. To my knowledge
there has only been a little prior work on prediction of the quality of the
outputs of a vision algorithm when ground truth is not available.
Jammalamadaka et al., (2012) discuss _evaluator algorithms_ to predict if the
output of a human pose estimation algorithm is correct or not, and Xia et al.,
(2020) have looked at predicting failures in semantic segmentation. In the
reconstructive framework the goodness-of-fit of the geometric variables
(camera parameters, object pose and shape) can be measured by the
intersection-over-union (IoU) of the the predicted and ground-truth object
masks (as used e.g. in Romaszko et al., 2020). Thus this IoU measure could be
predicted by an evaluator algorithm. Errors in these variables will have
consequent effects on the pixel errors. But if the IoU is satisfactory, then
object-level pixel errors will likely be due to errors in the texture or
lighting variables. Assessing the significance of such errors with an
evaluator algorithm will require the annotations of the severity and types of
the errors made.

It may be thought that algorithms that provide estimates of the _uncertainty_
in their predictions help to address the issue of the assessment of vision
algorithms, and this is partially correct. However, a limitation of
probabilistic model uncertainty is that it is assessed _relative_ to a model
(or a fixed set of models). But if, for example, the model’s palette of
textures does not match with that in an input image, then the model’s
posterior uncertainty measures will not characterize the true situation well.
This deficiency is known as the “open world” (M-open) situation, in contrast
to M-closed, as discussed e.g. by Bernardo and Smith, (1994, §6.1.2). One
approach to addressing this is via _model criticism_ , as discussed e.g., by
Seth et al., (2018).

Given the complexity of visual scenes, it may not be possible all the detail
in a scene at the pixel level. Instead, one can build a generative model of
the spatial layout of _image features_ , as computed e.g. by a DNN. Such an
approach is used, for example, in the neural mesh model (NeMo) of Wang et al.,
(2021). While it is more interpretable to reconstruct the input image than a
feature-based representation, the latter may make modelling easier. The
experimental results for the NeMo generative model show that it is much more
robust to OOD tasks like recognition under partial occlusion and prediction of
previously unseen poses than standard feedforward DNNs.

##  5 Advancing the SGM agenda

Above I have laid out the key topics of modelling objects and scenes, and
carrying out inference in these models. Below I comment on the state-of-the-
art, and issues around datasets etc.

##### Modelling objects:

The state-of-the-art seems to be at a good level. In terms of data, the Amazon
Berkeley Objects (ABO) dataset of Collins et al., (2022) is a recent example
of a reasonably large collection of models (some 8,000 objects) with complex
geometries and high-resolution, physically-based materials that allow for
photorealistic rendering.

##### Modelling scenes:

Compared to objects, the state-of-the-art is less advanced. Much of the focus
has been on indoor scenes, leaving the modelling of outdoor scenes in urban
and rural environments less explored. The task of modelling scenes is also
more difficult than for objects, involving variable numbers of objects and
types, and spatial, functional and semantic relationships between them. There
is still much to be done here, e.g., for automatic discovery of hierarchical
structure such as scene-type, objects and parts from data.

One issue here is data. Progress in 2D image recognition has been driven by
large-scale datasets. Currently there are very few large 3D datasets
available, and none on the scale of, say, the ImageNet which contains over one
million images. To my knowledge the 3D-FRONT dataset (Fu et al.,, 2021) is the
largest collection of indoor scenes, with almost 19,000 rooms. (It consists of
synthetic 3D indoor scenes with professionally designed layouts.) Of course
the effort needed to obtain and annotate a 3D scene is much greater than
simply annotating bounding boxes in images. But are there reasons to believe
that we should need fewer examples for learning from 3D data? Computer
graphics can, of course, provide a rich source of 3D ground truth and
annotations (see e.g. the CLEVR dataset of Johnson et al., 2017), and can
provide a controlled means to test compositional generalization (see their
sec. 4.7). However, this requires good object and scene models to generate
realistic scene layouts, and also there are issues on how well models trained
on such rendered data will transfer to real scenes.

While collecting 3D datasets is challenging, it is also possible to collect
_multi-view data_ , where there are multiple images of (parts of a) given
scene, with known camera parameters (intrinsic and extrinsic). See, e.g., the
active vision dataset from UNC161616
https://www.cs.unc.edu/~ammirato/active_vision_dataset_website/. and the Aria
dataset171717https://about.meta.com/realitylabs/projectaria/datasets/. from
Meta. SGMs can be tested on multi–view by predicting what will be observed
from a novel test viewpoint.

##### Inference:

Scene-level inference is challenging, with the need to match portions of a
hierarchical scene model with image data. As we have seen above, this can
involve bottom-up and top-down flows of information in the model. This could
lead to complex inference processes, reminiscent of those found in earlier
vision models, such as VISIONS (Hanson and Riseman,, 1978) and the system of
Ohta et al., (1978). It may be possible to simplify this somewhat by using the
idea of “lazy inference”, where only parts of the whole scene representation
are activated in response to given task. With regard to model deficiency, to
make progress it will be important to get lots of data on the kinds of
deficiencies that occur most, in order to address the important issues.

##### Tasks and Benchmarks:

Currently most tasks for computer vision are evaluated in the image plane.
Partly this may be due to the fact that it is relatively easy to collect
images and create annotations in this case. However, there are some 3D
benchmarks such as the KITTI suite181818http://www.cvlibs.net/datasets/kitti/.
and nuScenes191919https://www.nuscenes.org.. These include 3D object detection
in road scenes (using 3D bounding boxes). Autonomous driving and bin picking
tasks in cluttered scenes are examples of areas that may well push the 3D
reconstructive agenda forward.

In order to exploit the value of structured generative models, it will be
necessary to define a set of tasks which can exploit the same underlying
representation. Some examples of challenging tasks were given in sec. 1.3;
these include object reconstruction under heavy occlusion, and scene editing
tasks, both of which exploit the 3D and scene-level information contained in
the SGM representation. The research community will need to focus attention on
a tasteful choice of 3D/multi-view benchmarks and tasks in order to promote
the SGM approach.

## Acknowledgements

I thank David Hogg, Alan Yuille, Oisin Mac Aodha, Antonio Vergari, Siddharth
N., Kevin Murphy, Paul Henderson, Adam Kortylewski, Hakan Bilen and Titas
Anciukevicius for helpful comments and discussions.

For the purpose of open access, the author has applied a Creative Commons
Attribution (CC BY) licence to any Author Accepted Manuscript version arising
from this submission.

## References

  * Antol et al., (2015) Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C., and Parikh, D. (2015).  VQA: Visual Question Answering.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2425–2433.
  * Babalola et al., (2008) Babalola, K. O., Cootes, T. F., Twining, C. J., Petrovic, V., and Taylor, C. (2008).  3D Brain Segmentation Using Active Appearance Models and Local Regressors.  In Metaxas, D., Axel, L., Fichtinger, G., and Székely, G., editors, International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2008), pages 401–408. Springer.  Lecture Notes in Computer Science, vol 5241.
  * Bengio et al., (2009) Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009).  Curriculum Learning.  In Bottou, L. and Littman, M., editors, Proceedings of the Twenty-Sixth International Conference on Machine Learning (ICML 2009), pages 41–48.
  * Bernardo and Smith, (1994) Bernardo, J. M. and Smith, A. F. M. (1994).  Bayesian Theory.  Wiley, Chichester, UK.
  * Biederman, (1987) Biederman, I. (1987).  Recognition-by-components: A theory of human image understanding.  Psychological Review, 94:115–147.
  * Biederman et al., (1982) Biederman, I., Mezzanotte, R. J., and Rabinowitz, J. C. (1982).  Scene Perception: Detecting and Judging Objects Undergoing Relational Violations.  Cognitive Psychology, 14:143–177.
  * Blanz and Vetter, (1999) Blanz, V. and Vetter, T. (1999).  A Morphable Model For The Synthesis Of 3D Faces.  In SIGGRAPH 99: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pages 187–194. ACM Press.
  * Box and Draper, (1987) Box, G. E. P. and Draper, N. R. (1987).  Empirical Model-Building and Response Surfaces.  John Wiley & Sons.
  * Brodatz, (1966) Brodatz, P. (1966).  Textures: A Photographic Album for Artists and Designers.  Dover Publications, New York.
  * Charniak, (1993) Charniak, E. (1993).  Statistical Language Learning.  MIT Press, Cambridge, Massachusetts.
  * Collins et al., (2022) Collins, J., Goael, S., Deng, K., et al. (2022).  ABO: Dataset and Benchmarks for Real-World 3D Object Understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2022.
  * Cootes et al., (1998) Cootes, T. F., Edwards, G. J., and Taylor, C. J. (1998).  Active Appearance Models.  In Burkhardt, H. and Neumann, B., editors, Proceedings of the Fifth European Conference on Computer Vision, ECCV 1998, pages II 484–498. Springer.  Lecture Notes in Computer Science 1407.
  * Cootes et al., (1995) Cootes, T. F., Taylor, C. J., Cooper, D. H., and Graham, J. (1995).  Active Shape Models—Their Training and Application.  Computer Visoin and Image Understanding, 61(1):38–59.
  * Cover and Thomas, (1991) Cover, T. M. and Thomas, J. A. (1991).  Elements of Information Theory.  John Wiley & Sons.
  * Dayan et al., (1995) Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995).  The Helmholtz Machine.  Neural Computation, 7(5):889–904.
  * De Sousa Ribeiro et al., (2022) De Sousa Ribeiro, F., Duarte, K., Everett, M., Leontidis, G., and Shah, M. (2022).  Learning with Capsules: A Survey.  arXiv:2206.02664.
  * Efros and Leung, (1999) Efros, A. A. and Leung, T. K. (1999).  Texture Synthesis by Non-parametric Sampling.  In Proc International Conference on Computer Vision (ICCV 1999), pages 1033–1038.
  * Eslami et al., (2016) Eslami, S. M. A., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Kavukcuoglu, K., and Hinton, G. E. (2016).  Attend, infer, repeat: Fast scene understanding with generative models.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems.
  * Eslami and Williams, (2011) Eslami, S. M. A. and Williams, C. K. I. (2011).  Factored Shapes and Appearances for Parts-based Object Understanding.  In Proceedings of the British Machine Vision Conference 2011.
  * Eslami and Williams, (2012) Eslami, S. M. A. and Williams, C. K. I. (2012).  A Generative Model for Parts-based Object Segmentation.  In Bartlett, P., Pereira, F., Burges, C., Bottou, L., and Weinberger, K., editors, Advances in Neural Information Processing Systems 25, pages 100–107.
  * Everingham et al., (2010) Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. (2010).  The PASCAL Visual Object Classes (VOC) Challenge.  International Journal of Computer Vision, 88(2):303–338.
  * Felzenszwalb et al., (2009) Felzenszwalb, P., Girshick, R., McAllester, D., and Ramanan, D. (2009).  Object Detection with Discriminatively Trained Part Based Models.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627–1645.
  * Fischler and Bolles, (1981) Fischler, M. A. and Bolles, R. C. (1981).  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395.
  * Fischler and Elschlager, (1973) Fischler, M. A. and Elschlager, R. A. (1973).  The Representation and Matching of Pictoral Structures.  IEEE Transactions on Computers, 22(1):67–92.
  * Forsyth and Ponce, (2003) Forsyth, D. A. and Ponce, J. (2003).  Computer Vision: A Modern Approach.  Prentice Hall, second edition.
  * Frey and Jojic, (2003) Frey, B. J. and Jojic, N. (2003).  Transformation Invariant Clustering Using the EM Algorithm.  IEEE Trans Pattern Analysis and Machine Intelligence, 25(1):1–17.
  * Fu et al., (2021) Fu, H., Cai, B., Gao, L., Zhang, L.-X., Wang, J., Li, C., Zeng, Q., Sun, C., Jia, R., Zhao, B., and Zhang, H. (2021).  3D-FRONT: 3D furnished rooms with layouts and semantics.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933–10942.
  * Fu, (1982) Fu, K. S. (1982).  Syntactic Pattern Recognition and Applications.  Prentice-Hall.
  * Galerne et al., (2011) Galerne, B., Gousseau, Y., and Morel, J.-M. (2011).  Random Phase Textures: Theory and Synthesis.  IEEE Transactions on Image Processing, 20(1):257–267.
  * Greff et al., (2019) Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A. (2019).  Multi-Object Representation Learning with Iterative Variational Inference.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 2424–2433.
  * Grenander, (1978) Grenander, U. (1978).  Lectures in Pattern Theory: Volume 2 Pattern Analysis.  Springer-Verlag.
  * Hall, (1973) Hall, P. A. V. (1973).  Equivalence Between AND/OR Graphs and Context-Free Grammars.  Communications of the ACM, 16(7):444–445.
  * Hanson and Riseman, (1978) Hanson, A. R. and Riseman, E. M. (1978).  VISIONS: A Computer System for Interpreting Scenes.  In Hanson, A. R. and Riseman, E. M., editors, Computer Vision Systems. Academic Press.
  * Heitz and Koller, (2008) Heitz, G. and Koller, D. (2008).  Learning Spatial Context: Using Stuff to Find Things.  In Proc ECCV 2008.
  * Hinton, (2002) Hinton, G. E. (2002).  Training Products of Experts by Minimizing Contrastive Divergence.  Neural Computation, 14:1771–1800.
  * Hinton et al., (2011) Hinton, G. E., Krizhevsky, A., and Wang, S. D. (2011).  Transforming Auto-encoders.  In Proceedings ICANN 2011.
  * Hinton et al., (2018) Hinton, G. E., Sabour, S., and Frosst, N. (2018).  Matrix capsules with EM routing.  In International Conference on Learning Representations.
  * Hueting et al., (2018) Hueting, M., Reddy, P., Kim, V. G., Yumer, E., Carr, N., and Mitra, N. J. (2018).  Seethrough: Finding objects in heavily occluded indoor scene images.  In Proceedings of International Conference on 3DVision (3DV).
  * Izadinia et al., (2017) Izadinia, H., Shan, Q., and Setiz, S. M. (2017).  IM2CAD.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017.
  * Jammalamadaka et al., (2012) Jammalamadaka, N., Zisserman, A., Eichner, M., Ferrari, V., and Jawahar, C. (2012).  Has My Algorithm Succeeded? An Evaluator for Human Pose Estimators.  In Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., and Schmid, C., editors, Computer Vision—ECCV 2012. Springer.  Lecture Notes in Computer Science 7574.
  * Jang and Agapito, (2021) Jang, W. and Agapito, L. (2021).  CodeNeRF: Disentangled Neural Radiance Fields for Object Categories.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12949–12958.  Also available as arXiv:2109.01750.
  * Jin and Geman, (2006) Jin, Y. and Geman, S. (2006).  Context and Hierarchy in a Probabilistic Image Model.  In Proc CVPR 2006. IEEE Computer Society.
  * Johnson et al., (2017) Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. (2017).  CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning.  In CVPR, pages 1988–1997.
  * Jordan et al., (1999) Jordan, M. I., Ghahramani, Z., and Jaakkola, T. S. Saul, L. K. (1999).  An Introduction to Variational Methods for Graphical Models.  Machine Learning, 37:183–233.
  * Kato et al., (2020) Kato, H., Beker, D., Morariu, M., Ando, T., Matsuoka, T., Kehl, W., and Gaidon, A. (2020).  Differentiable Rendering: A Survey.  https://arxiv.org/2006.12057.
  * Kim et al., (2019) Kim, Y., Dyer, C., and Rush, A. (2019).  Compound Probabilistic Context-Free Grammars for Grammar Induction.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2369–2385.
  * Kingma and Welling, (2014) Kingma, D. P. and Welling, M. (2014).  Auto-Encoding Variational Bayes.  In ICLR.
  * Kivinen and Williams, (2012) Kivinen, J. J. and Williams, C. K. I. (2012).  Multiple Texture Boltzmann Machines.  In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics.
  * Kosiorek et al., (2019) Kosiorek, A., Sabour, S., Teh, Y. W., and Hinton, G. E. (2019).  Stacked Capsule Autoencoders.  In Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32.
  * Kulkarni et al., (2015) Kulkarni, T. D., Whitney, W. F., Kohli, P., and Tenenbaum, J. (2015).  Deep convolutional inverse graphics network.  In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 28.
  * Land et al., (1999) Land, M., Mennie, N., and Rusted, J. (1999).  The roles of vision and eye movements in the control of activities of daily living.  Perception, 28:1311–1328.
  * Lee et al., (2009) Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. (2009).  Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations.  In Bottou, L. and Littman, M., editors, Proceedings of the Twenty-Sixth International Conference on Machine Learning (ICML 2009).
  * Lee et al., (2019) Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., and Teh, Y.-W. (2019).  Set Transformer: A Framework for Attention-based Permutation-invariant Neural Networks.  In Proc. of the 36th International Conference on Machine Learning, pages 3744–3753.
  * Li et al., (2019) Li, M., Gadi Patil, A., Xu, K., Chaudhuri, S., Khan, O., Shamir, A., Tu, C., Chen, B., Cohen-Or, D., and Zhang, H. (2019).  GRAINS: Generative Recursive Autoencoders for Indoor Scenes.  ACM Transactions on Graphics, 37(2):1–16.
  * Li et al., (2020) Li, N., Eastwood, C., and Fisher, R. (2020).  Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 5656–5666.
  * Liu et al., (2014) Liu, T., Chaudhuri, S., Kim, V., Huang, Q.-X., Mitra, N. J., and Funkhouser, T. (2014).  Creating Consistent Scene Graphs Using a Probabilistic Grammar.  ACM Transactions on Graphics (Proc. of SIGGRAPH Asia), 33(6).
  * Locatello et al., (2020) Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. (2020).  Object-Centric Learning with Slot Attention.  In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 11525–11538.
  * Loper et al., (2015) Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. (2015).  SMPL: a Skinned Multi-person Linear Model.  ACM Transactions on Graphics, 34(6).
  * Loper and Black, (2014) Loper, M. M. and Black, M. J. (2014).  OpenDR: An Approximate Differentiable Renderer.  In Computer Vision-ECCV 2014, pages 154–169.
  * MacKay, (2003) MacKay, D. J. C. (2003).  Information Theory, Inference, and Learning Algorithms.  Cambridge University Press, Cambridge.
  * Marcus et al., (2022) Marcus, G., Davis, E., and Aaronson, S. (2022).  A very preliminary analysis of DALL-E 2.  arXiv:2204.13807.
  * Mildenhall et al., (2020) Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. (2020).  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J., editors, Computer Vision–ECCV 2020. Springer.  Lecture Notes in Computer Science 12346.
  * Moreno et al., (2016) Moreno, P., Williams, C. K. I., Nash, C., and Kohli, P. (2016).  Overcoming occlusion with inverse graphics.  In Gang, H. and Jegou, H., editors, Computer Vision-ECCV 2016 Workshops Proceedings Part III, pages 170–185. Springer.  LNCS 9915.
  * Murphy, (2012) Murphy, K. P. (2012).  Machine Learning: a Probabilistic Perspective.  MIT Press.
  * Nazabal et al., (2022) Nazabal, A., Tsagkas, N., and Williams, C. K. I. (2022).  Inference and Learning for Generative Capsule Models.  arXiv:2209.03115.
  * Ohta et al., (1978) Ohta, Y., Kanade, T., and Sakai, T. (1978).  An Analysis System for Scenes Containing Objects with Substructures.  In Proc. 4th IJCPR, pages 752–754.
  * Oliva and Torralba, (2006) Oliva, A. and Torralba, A. (2006).  Building the gist of a scene: the role of global image features in recongition.  Progress in Brain Research, 155:23–36.
  * Palmer, (1999) Palmer, S. E. (1999).  Vision Science: Photons to Phenomenology.  MIT Press.
  * Paschalidou et al., (2021) Paschalidou, D., Kar, A., Shugrina, M., Kreis, K., Geiger, A., and Fidler, S. (2021).  ATISS: Autoregressive Transformers for Indoor Scene Synthesis.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 12013–12026. Curran Associates, Inc.
  * Prince, (2012) Prince, S. J. D. (2012).  Computer vision: models, learning and inference.  Cambridge University Press.
  * Quinn et al., (2009) Quinn, J. A., Williams, C. K. I., and McIntosh, N. (2009).  Factorial Switching Linear Dynamical Systems applied to Physiological Condition Monitoring.  IEEE Trans. on Pattern Analysis and Machine Intelligence, 31(9):1537–1551.
  * Radford et al., (2016) Radford, A., Metz, L., and Chintala, S. (2016).  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.  In International Conference on Learning Representations (ICLR 2016).
  * Ramesh et al., (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021).  Zero-shot text-to-image generation.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821–8831. PMLR.
  * Ranzato et al., (2010) Ranzato, M., Mnih, V., and Hinton, G. E. (2010).  Generating more realistic images using gated MRF’s.  In Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., and Culotta, A., editors, Advances in Neural Information Processing Systems, volume 23.
  * Rasmussen and Williams, (2006) Rasmussen, C. E. and Williams, C. K. I. (2006).  Gaussian Processes for Machine Learning.  MIT Press, Cambridge, Massachusetts.
  * Rezende et al., (2014) Rezende, D. J., Mohamed, S., and Wierstra, D. (2014).  Stochastic Backpropagation and Approximate Inference in Deep Generative Models.  In ICML.
  * Ritchie, (2019) Ritchie, D. (2019).  Generative Models of 3D Scenes.  Part of the tutorial on Learning Generative Models of 3D Structures at Eurographics 2019, https://3dstructgen.github.io.
  * Ritchie et al., (2019) Ritchie, D., Wang, K., and Lin, Y.-A. (2019).  Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models.  In CVPR 2019.
  * Roberts, (1963) Roberts, L. G. (1963).  Machine Perception of Three-dimensional Solids.  Technical Report 315, MIT Lincoln Laboratory.
  * Romaszko et al., (2020) Romaszko, L., Williams, C. K. I., and Winn, J. (2020).  Learning Direct Optimization for Scene Understanding.  Pattern Recognition, 105:107369.
  * Ross and Zemel, (2006) Ross, D. and Zemel, R. (2006).  Learning parts-based representations of data.  Journal of Machine Learning Research, 7:2369–2397.
  * Roth and Black, (2005) Roth, S. and Black, M. J. (2005).  Fields of Experts: A Framework for Learning Image Priors.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2005, pages III:860–867.
  * Rozanov, (1977) Rozanov, J. A. (1977).  Markov Random Fields and Stochastic Partial Differential Equations.  Math. USSR Sbornik, 32(4):515–534.
  * Rünz et al., (2020) Rünz, M., Li, K., Tang, M., Ma, L., Kong, C., Schmidt, T., Reid, I., Agapito, L., Straub, J., Lovegrove, S., and Newcombe, R. (2020).  FroDO: From Detections to 3D Objects.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2020).
  * Sabour et al., (2017) Sabour, S., Frosst, N., and Hinton, G. (2017).  Dynamic routing between capsules.  In Advances in Neural Information Processing Systems, pages 3856–3866.
  * Salimans et al., (2017) Salimans, T., Karpathy, A., Chen, X., Kingma, D. P., and Bulatov, Y. (2017).  PixelCNN++: A PixelCNN Implementation with Discretized Logistic Mixture Likelihood and Other Modifications.  In Proceedings of the International Conference on Learning Representations (ICLR 2017).
  * Schönberger and Frahm, (2016) Schönberger, J. L. and Frahm, J.-M. (2016).  Structure-from-motion revisited.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016).
  * Seth et al., (2018) Seth, S., Murray, I., and Williams, C. K. I. (2018).  Model Criticism in Latent Space.  Bayesian Analysis , 14(3):703–725.
  * Shi et al., (2017) Shi, M., Caesar, H., and Ferrari, V. (2017).  Weakly Supervised Object Localization Using Things and Stuff Transfer.  In Proc. International Conference on Computer Vision (ICCV 2017).
  * Sun et al., (2014) Sun, M., Kim, B., Kohli, P., and Savarese, S. (2014).  Relating Things and Stuff via Object Property Interactions.  IEEE Trans Pattern Analysis and Machine Intelligence, 36(7):1370–1383.
  * Szegedy et al., (2013) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013).  Intriguing properties of neural networks.  arXiv:1312.6199.
  * Szeliski, (2021) Szeliski, R. (2021).  Computer Vision: Algorithms and Applications.  Springer, second edition.
  * Tenenbaum and Freeman, (2000) Tenenbaum, J. B. and Freeman, W. T. (2000).  Separating style and content with bilinear models.  Neural Computation, 12:1247–1283.
  * Torralba et al., (2010) Torralba, A., Murphy, K. P., and Freeman, W. T. (2010).  Using the Forest to See the Trees: Exploiting Context for Visual Object Detection and Localization.  Communications of the ACM, 53(3):107–114.
  * Tu and Zhu, (2002) Tu, Z. and Zhu, S.-C. (2002).  Image Segmentation by Data-Driven Markov Chain Monte Carlo.  IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(5):657–673.
  * van den Burg and Williams, (2021) van den Burg, G. J. J. and Williams, C. K. I. (2021).  On memorization in probabilistic deep generative models.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 27916–27928. Curran Associates, Inc.
  * van den Oord et al., (2016) van den Oord, A., Kalchbrenner, N., Espeholt, L., kavukcuoglu, k., Vinyals, O., and Graves, A. (2016).  Conditional Image Generation with PixelCNN Decoders.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems.
  * Vasilescu and Terzopoulos, (2002) Vasilescu, M. A. O. and Terzopoulos, D. (2002).  Multilinear Analysis of Image Ensembles: TensorFaces.  In Heyden, A., Sparr, G., Nielsen, M., and Johansen, P., editors, Computer Vision–ECCV 2002. Springer.  Lecture Notes in Computer Science 2350.
  * Wang et al., (2021) Wang, A., Kortylewski, A., and Yuille, A. (2021).  NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation.  In Proceedings of the International Conference on Learning Representations (ICLR 2021).
  * Williams and Titsias, (2004) Williams, C. K. I. and Titsias, M. K. (2004).  Greedy Learning of Multiple Objects in Images using Robust Statistics and Factorial Learning.  Neural Computation, 16(5):1039–1062.
  * Xia et al., (2020) Xia, Y., Zhang, Y., Liu, F., Shen, W., and Yuille, A. (2020).  Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation.  In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J., editors, Computer Vision—ECCV 2020. Springer.  Lecture Notes in Computer Science 12346.
  * Yao et al., (2007) Yao, B., Yang, X., and Zhu, S.-C. (2007).  Introduction to a Large-Scale General Purpose Ground Truth Database: Methodology, Annotation Tool and Benchmarks.  In EMMCVPR’07: Proceedings of the 6th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition.
  * Yeh et al., (2012) Yeh, Y.-T., Yang, L., Watson, M., Goodman, N. D., and Hanrahan, P. (2012).  Synthesizing open worlds with constraints using locally annealed reversible jump MCMC.  ACM Transactions on Graphics, 31(4):1–11.
  * Yu et al., (2011) Yu, L.-F., Yeung, S.-K., Tang, C.-K., Terzopoulos, D., Chan, T. F., and Osher, S. J. (2011).  Make it home: automatic optimization of furniture arrangement.  ACM Transactions on Graphics, 30(4).
  * Yuille and Liu, (2021) Yuille, A. L. and Liu, C. (2021).  Deep Nets: What have they ever done for Vision?  International Journal of Computer Vision, 129:781–802.
  * Zamir et al., (2020) Zamir, A. R., Sax, A., Yeo, T., Kar, O., Cheerla, N., Suri, R., Cao, Z., Mailk, J., and Guibas, L. (2020).  Robust Learning Through Cross-Task Consistency.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2020.
  * Zhang et al., (2021) Zhang, X., Srinivasan, P. P., Deng, B., Debevec, P., Freeman, W. T., and Barron, J. T. (2021).  NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination.  ACM Transactions on Graphics (Proc. of SIGGRAPH Asia), 40(6).
  * Zhu and Huang, (2021) Zhu, S.-C. and Huang, S. (2021).  Computer Vision: Stochastic Grammars for Parsing Objects, Scenes, and Events.  Springer.
  * Zhu and Mumford, (2006) Zhu, S.-C. and Mumford, D. (2006).  A Stochastic Grammar of Images.  Foundations and Trends in Computer Graphics and Vision, 2(4):259–362.
  * Zhu et al., (1998) Zhu, S.-C., Wu, Y., and Mumford, D. (1998).  Filters, Random Fields and Maximum Entropy (FRAME): Towards a unified theory for texture modeling.  Int. J. Comput. Vision, 27(2):107–126.