# Systematic Visual Reasoning through
Object-Centric Relational Abstraction

Taylor W. Webb*$\dagger$
Department of Psychology
University of California, Los Angeles
Los Angeles, CA
<EMAIL_ADDRESS>
Shanka Subhra Mondal*$\dagger$
Department of Electrical and Computer Engineering
Princeton University
Princeton, NJ
<EMAIL_ADDRESS>
* Equal Contribution
Jonathan D. Cohen
Princeton Neuroscience Institute
Princeton University
Princeton, NJ
<EMAIL_ADDRESS>

###### Abstract

Human visual reasoning is characterized by an ability to identify abstract
patterns from only a small number of examples, and to systematically
generalize those patterns to novel inputs. This capacity depends in large part
on our ability to represent complex visual inputs in terms of both objects and
relations. Recent work in computer vision has introduced models with the
capacity to extract object-centric representations, leading to the ability to
process multi-object visual inputs, but falling short of the systematic
generalization displayed by human reasoning. Other recent models have employed
inductive biases for relational abstraction to achieve systematic
generalization of learned abstract rules, but have generally assumed the
presence of object-focused inputs. Here, we combine these two approaches,
introducing Object-Centric Relational Abstraction (OCRA), a model that
extracts explicit representations of both objects and abstract relations, and
achieves strong systematic generalization in tasks (including a novel dataset,
CLEVR-ART, with greater visual complexity) involving complex visual displays.

##  1 Introduction

When presented with a visual scene, human reasoners have a capacity to
identify not only the objects in that scene, but also the relations between
objects, and the higher-order patterns formed by multiple relations. These
abilities are fundamental to human visual intelligence, enabling the rapid
induction of abstract rules from a small number of examples, and the
systematic generalization of those rules to novel inputs [32, 25, 13, 27]. For
example, when presented with the image in Figure 1, one can easily determine
that the objects in the top and bottom rows are both governed by a common
abstract pattern (ABA). Furthermore, one could easily then generalize this
pattern to any potential shapes, even those that one has never observed
before. Indeed, this is arguably what it means for a pattern to be considered
abstract.

Though this task appears effortless from the perspective of human reasoning,
it exemplifies the type of problems that pose a significant challenge for
standard neural network methods. Rather than inducing an abstract
representation of the underlying rules in a given task, neural networks tend
to overfit to the specific details of the examples observed during training,
thus failing to generalize those rules to novel inputs [26, 3, 33].

Figure 1: Object-Centric Relational Abstraction (OCRA). OCRA consists of three
core components, responsible for modeling objects, relations, and higher-order
relations. First, given a visual input $\boldsymbol{x}$, slot attention
extracts object-centric representations, consisting of factorized feature
embeddings $\boldsymbol{z}_{k=1...K}$ and position embeddings
$\boldsymbol{m}_{k=1...K}$. Second, pairwise relation embeddings
$\boldsymbol{r}_{kk^{\prime}}$ are computed by first passing each pair of
feature embeddings to a relational operator $\phi$, and then additively
incorporating their position embeddings. This allows visual space to serve as
an indexing mechanism for visual relations. Finally, relational embeddings are
passed to a transformer to model the higher-order patterns formed by multiple
relations.

To address this, a number of neural network methods have recently been
proposed that incorporate strong inductive biases for relational abstraction
[49, 51, 22, 1], also referred to as a ‘relational bottleneck’ [50]. The key
insight underlying these approaches is that abstract visual patterns are often
characterized by the relations between objects, rather than the features of
the objects themselves. For example, the pattern seen in Figure 1 is
characterized by the relation between the shape and position of the objects.
By constraining architectures to process visual inputs in terms of relations
between objects, these recent approaches have achieved strong systematic
(i.e., out-of-distribution) generalization of learned abstract rules, given
only a small number of training examples [51, 22, 1].

However, a major assumption of these approaches is that relations are computed
between objects. Accordingly, these methods have generally relied on inputs
consisting of pre-segmented visual objects, rather than multi-object scenes.
Furthermore, these methods can sometimes be brittle to task-irrelevant visual
variation, such as spatial jitter or Gaussian noise [43]. It is thus unclear
whether and how these approaches might scale to complex and varied real-world
settings.

In this work, we take a step toward the development of relational reasoning
algorithms that can operate over more realistic visual inputs. To do so, we
turn to object-centric representation learning [15, 6, 28, 12], a recently
developed approach in which visual scenes are decomposed into a discrete set
of object-focused latent representations, without the need for veridical
segmentation data. Though this approach has shown success on a number of
challenging visual tasks [10, 39, 30], it has not yet been integrated with the
strong relational inductive biases that have previously been shown to enable
systematic, human-like generalization [49, 51, 22, 1]. Here, we propose
Object-Centric Relational Abstraction (OCRA), a model that combines the
strengths of these two approaches. Across a suite of visual reasoning tasks
(including a novel dataset, CLEVR-ART, with greater visual complexity), we
find that OCRA is capable of systematically generalizing learned abstract
rules from complex, multi-object visual displays, significantly outperforming
a number of competitive baselines in most settings.

##  2 Approach

Figure 1 shows a schematic depiction of our approach. OCRA consists of three
core components, each responsible for a different element of the reasoning
process. These three elements can be summarized as 1) extraction of object-
centric representations, 2) computation of pairwise relational embeddings, and
3) processing of higher-order relational structures – patterns across multiple
relations – in order to identify abstract rules. We describe each of these
three components in the following sections.

###  2.1 Object-centric representation learning

OCRA employs slot attention [28] to extract object-centric representations
from complex, multi-object visual inputs. Given an image $\boldsymbol{x}$, the
goal of slot attention is to extract a set of latent embeddings (i.e., slots),
each of which captures a focused representation of a single visual object,
typically without access to ground truth segmentation data.

To do so, the image is first passed through a series of convolutional layers,
generating a feature map $\boldsymbol{feat}\in\mathbb{R}^{H\times W\times D}$.
Positional embeddings $\boldsymbol{pos}\in\mathbb{R}^{H\times W\times D}$ are
then generated by passing a 4d position code (encoding the four cardinal
directions) through a separate linear projection. Feature and position
embeddings are additively combined, and then passed through a series of 1x1
convolutions. This is then flattened, yielding
$\boldsymbol{inputs}\in\mathbb{R}^{N\times D}$ (where $N=H\times W$), over
which slot attention is performed.

The set of $K$ $\boldsymbol{slots}\in\mathbb{R}^{K\times D}$ is then randomly
initialized (from a shared distribution with learned parameters), and a form
of transformer-style cross-attention is employed to attend over the pixels of
the feature map. Specifically, each slot emits a query
$\operatorname{q}(\boldsymbol{slots})\in\mathbb{R}^{K\times D}$ (through a
linear projection), and each location in the feature map emits a key
$\operatorname{k}(\boldsymbol{inputs})\in\mathbb{R}^{N\times D}$ and a value
$\operatorname{v}(\boldsymbol{inputs})\in\mathbb{R}^{N\times D}$. For each
slot, a query-key attention operation is then used to generate an attention
distribution over the feature map
$\boldsymbol{attn}=\operatorname{softmax}(\frac{1}{\sqrt{D}}\operatorname{k}(\boldsymbol{inputs})\cdot\operatorname{q}(\boldsymbol{slots})^{\top})$
and a weighted mean of the values
$\boldsymbol{updates}=\boldsymbol{attn}\cdot\operatorname{v}(\boldsymbol{inputs})$
is used to update the slot representations using a Gated Recurrent Unit [8],
followed by a residual MLP. This process is repeated for $T$ iterations,
leading to a competition between the slots to account for each pixel in the
feature map.

For the purposes of computing relational embeddings, we employed a factorized
object representation format, in which position is represented distinctly from
other object features. For a given input image, after performing $T$
iterations of slot attention, we used the final attention map
$\boldsymbol{attn}_{T}\in\mathbb{R}^{K\times N}$ to compute feature- and
position-specific embeddings for each slot:

$\boldsymbol{z}_{k}=\boldsymbol{attn}_{T}\operatorname{flatten}(\boldsymbol{feat})[k]$
(1)

$\boldsymbol{m}_{k}=\boldsymbol{attn}_{T}\operatorname{flatten}(\boldsymbol{pos})[k]$
(2)

where $\boldsymbol{feat}$ and $\boldsymbol{pos}$ represent the feature- and
position-embedding maps (prior to being combined). As will be seen in the
following section, this factorized representation allows visual space to serve
as an indexing mechanism for pairwise relations, mirroring the central role of
spatial organization in human visual reasoning [29].

Although slot attention can in principle be learned end-to-end in the service
of a specific downstream reasoning task, here we chose to pre-train slot
attention using an unsupervised objective. This decision reflects the
assumption that human-like visual reasoning does not occur in a vacuum, and
visual representations are generally not learned from scratch when performing
each new task. Instead, visual object representations are shaped by a wealth
of non-task-specific prior experience. To model the effects of this prior
experience, we pre-trained slot attention to reconstruct randomly generated
multi-object displays, using a slot-based autoencoder framework (more details
in Section 4.3.1). Importantly, for experiments testing generalization of
abstract rules to novel objects, we ensured that these objects did not appear
in the pre-training data for slot attention.

###  2.2 Relational embeddings

After extracting object embeddings, we employed a novel relational embedding
method, designed to allow OCRA to abstract over the perceptual features of
individual objects (and thus systematically generalize learned rules to inputs
with novel perceptual features). This embedding method consisted of two steps.
First, we applied a relational operator $\phi$ to compute the pairwise
relation between two feature embeddings $\boldsymbol{z}_{k}$ and
$\boldsymbol{z}_{k^{\prime}}$. In the present work, we define this operator
as:

$\phi(\boldsymbol{z}_{k},\boldsymbol{z}_{k^{\prime}})=(\boldsymbol{z}_{k}\boldsymbol{W_{z}}\cdot\boldsymbol{z}_{k^{\prime}}\boldsymbol{W_{z}})\boldsymbol{W_{r}}$
(3)

where $\boldsymbol{z}_{k}$ and $\boldsymbol{z}_{k^{\prime}}$ are first
projected through a shared set of linear weights
$\boldsymbol{W_{z}}\in\mathbb{R}^{D\times D}$, and the dot product between
these weighted embeddings is then projected back out into the original
dimensionality using another set of weights
$\boldsymbol{W_{r}}\in\mathbb{R}^{1\times D}$. Using dot products to model
pairwise relations introduces a layer of abstraction between the perceptual
features of the objects and the downstream reasoning process, in that the dot
product only represents the relations between objects, rather than those
objects’ individual features. We hypothesized that this relational bottleneck
would lead to improved generalization to inputs with novel perceptual
features. Additionally, the shared projection weights ($\boldsymbol{W_{z}}$)
enable the model to learn which features are relevant for a given task, and to
only compute relations between those features.

After applying the relational operator, we then compute the relational
embedding for objects $k$ and $k^{\prime}$ as:

$\boldsymbol{r}_{kk^{\prime}}=\phi(\boldsymbol{z}_{k},\boldsymbol{z}_{k^{\prime}})+\boldsymbol{m}_{k}\boldsymbol{W_{m}}+\boldsymbol{m}_{k^{\prime}}\boldsymbol{W_{m}}$
(4)

where the position embeddings $\boldsymbol{m}_{k}$ and
$\boldsymbol{m}_{k^{\prime}}$ are each projected through a shared set of
linear weights111Note that the linear nature of this transformation is
important: a nonlinear transformation of the position embeddings (e.g., via a
MLP) could result in the extraction of implicit shape information, and
therefore break the relational bottleneck.
$\boldsymbol{W_{m}}\in\mathbb{R}^{D\times D}$ and added to the output of
$\phi$. This allows the relation between objects $k$ and $k^{\prime}$ to be
indexed by their spatial position, thus endowing OCRA with an explicit
variable-binding mechanism.

###  2.3 Processing of higher-order relations

After computing all $K^{2}$ pairwise relational embeddings, we then pass these
embeddings to a reasoning process to extract higher-order relations – patterns
formed between multiple relations. This is an essential component of visual
reasoning, as abstract rules, such as the one displayed in Figure 1, are
typically defined in terms of higher-order relations. We model these higher-
order interactions using a transformer [45]:

$\boldsymbol{y}=\operatorname{Transformer}(\boldsymbol{r}_{k=1,k^{\prime}=1}...\boldsymbol{r}_{k=K,k^{\prime}=K})$
(5)

where the inputs are formed by the set of all pairwise relational embeddings
$\boldsymbol{r}_{kk^{\prime}}$ (for all $k=1...K$ and all $k^{\prime}=1...K$),
and $\boldsymbol{y}$ is a task-specific output. Because we employ a symmetric
relational operator in the present work, we do not in practice present the
full matrix of pairwise comparisons, but instead limit these comparisons to
those in the upper triangular matrix.

##  3 Related Work

A number of methods for annotation-free object segmentation have recently been
proposed [15, 6, 28, 12, 11, 20], including the slot attention method employed
by our proposed model [28]. Though the details of these methods differ, they
share the general goal of decomposing visual scenes into a set of latent
variables corresponding to objects, without access to ground truth
segmentation data. Building on the success of these object-encoding methods,
some recent work has developed object-centric approaches to visual reasoning
that combine slot-based object representations with transformer-based
architectures [10, 39, 52, 30]. This work has yielded impressive performance
on challenging computer vision tasks, such as question answering from video
[10], or complex multi-object visual analogy problems [30], but has thus far
been limited to generalization to similar problem types following extensive
task-specific training.

In the present work, our goal was to develop a model capable of human-like
systematic generalization in abstract reasoning tasks. Our proposed approach
builds on the insights of other recent work in this area, most notably
including the Emergent Symbol Binding Network (ESBN) [51] and CoRelNet [22]
architectures, both of which employed the concept of a relational bottleneck
in different ways. In the ESBN architecture, separate control and perception
pathways interact indirectly through a key-value memory, allowing the control
pathway to learn representations that are abstracted over the details
represented in the perceptual pathway. Kerg et al. [22] proposed that the
ESBN’s capacity for abstraction was due to the implicit encoding of relational
information in its memory operations, which are mediated by a dot-product-
based retrieval operation. Kerg et al. also proposed a novel architecture,
CoRelNet, that makes this inductive bias more explicit. The CoRelNet
architecture operates by explicitly computing the full matrix of relations
(based on dot products) between each pair of objects in its input, and then
passing these relations to an MLP for further processing. Both ESBN and
CoRelNet displayed strong systematic generalization of learned abstract rules
from a small set of examples. Importantly, however, both of these
architectures depended on the presence of pre-segmented objects as input. A
primary contribution of the present work is to combine these inductive biases
for relational abstraction (i.e., the relational bottleneck) with object-
centric encoding mechanisms, and to introduce the use of a spatial indexing
mechanism that allows the model to apply these inductive biases to spatially
embedded, multi-object visual inputs.

In addition to lacking object encoding mechanisms, it was recently shown that
the ESBN can sometimes be overly brittle to task-irrelevant visual variation.
Vaishnav and Serre [43] evaluated the ESBN and other reasoning architectures
on abstract visual reasoning problems similar to those evaluated by Webb et
al. [51], but with the introduction of either spatial jitter or Gaussian
noise, finding that this significantly impaired performance in the ESBN (as
well as other reasoning architectures, such as the transformer [45] and
relation net [35]). Vaishnav and Serre also introduced a novel architecture,
GAMR, that was more robust to these sources of noise, and able to solve
problems based on multi-object inputs, likely due to the use of a novel guided
attention mechanism. But despite outperforming ESBN and other architectures in
this more challenging visual setting, GAMR nevertheless fell short of
systematic generalization in these tasks, displaying a significant drop in
performance in the most difficult generalization regimes. Here, we directly
confront this challenge, demonstrating that OCRA is capable of systematic
generalization comparable to that displayed by ESBN and CoRelNet, but in more
complex visual settings.

It is also worth mentioning that other neural network architectures have been
proposed that incorporate relational inductive biases in various ways [35, 4,
38, 34, 37, 5], some of which also incorporate object-centric representations
[48, 44, 19, 53, 2, 24, 40, 46, 14, 41]. Relative to this previous work, our
approach is distinguished by the inclusion of a stronger inductive bias toward
relational abstraction – that is, a relational bottleneck – for promoting the
development of relational representations that fully abstract over the details
of individual objects [51, 22, 1]. To empirically evaluate the importance of
the relational bottleneck, we compare our approach with a set of baseline
models that capture the key computational properties of previous models,
including a slot-transformer baseline that combines slot attention with a
transformer reasoning module [10, 39, 52, 30], and a slot-interaction-network
baseline that combines slot attention with an interaction-network reasoning
module [48, 44, 19, 53, 2, 24, 40, 46, 14, 41].

To summarize our contributions relative to previous work, our proposed model
includes:

  1. 1.

A novel object representation format that is factorized into distinct feature
and position embeddings (Equations 1 and 2), enabling a form of explicit
variable-binding.

  2. 2.

A novel relational embedding method that implements a relational bottleneck
(Equations 3 and 4).

  3. 3.

An architecture that combines these elements with preexisting components (slot
attention and transformers) in a novel manner to support systematic visual
reasoning from complex visual displays, including a novel dataset CLEVR-ART
with greater visual complexity.

##  4 Experiments

###  4.1 Datasets

We evaluated OCRA on three challenging visual reasoning tasks, ART [51] and
SVRT [13], specifically designed to probe systematic generalization of learned
abstract rules, as well as a novel dataset, CLEVR-ART, involving more complex
visual inputs (Figure 2).

Figure 2: Abstract visual reasoning tasks. Three datasets used to evaluate
systematic visual reasoning. Abstract Reasoning Tasks (ART) consist of four
separate visual reasoning tasks: same/different, relational-match-to-sample,
distribution-of-3, and identity rules (pictured). Systematic generalization is
studied by training on problems involving a small number of objects (as few as
5), and testing on heldout objects. Synthetic Visual Reasoning Test (SVRT)
consists of 23 tasks, each defined by separate abstract rule. Rules are
defined based on either spatial relations (SR) or same/different relations
(SD; pictured). In example problem, category 1 always contains three sets of
two identical objects, category 2 always contains two sets of three identical
objects. CLEVR-ART is a novel dataset consisting of ART problems rendered
using more realistic 3D shapes, based on the CLEVR dataset.

ART. The Abstract Reasoning Tasks (ART) dataset was proposed by Webb et al.
[51], consisting of four visual reasoning tasks, each defined by a different
abstract rule (Figure S1). In the ‘same/different’ task, two objects are
presented, and the task is to say whether they are the same or different. In
the ‘relational match-to-sample’ task, a source pair of objects is presented
that either instantiates a ‘same’ or ‘different’ relation, and the task is to
select the pair of target objects (out of two pairs) that instantiates the
same relation. In the ‘distribution-of-3’ task, a set of three objects is
presented in the first row, and an incomplete set is presented in the second
row. The task is to select the missing object from a set of four choices. In
the ‘identity rules’ task, an abstract pattern is instantiated in the first
row (ABA, ABB, or AAA), and the task is to select the choice that would result
in the same relation being instantiated in the second row.

The primary purpose of this dataset was to evaluate strong systematic
generalization of these rules. Thus, each task consists of a number of
generalization regimes of varying difficulty, defined by the number of unique
objects used to instantiate the rules during training (out of a set of 100
possible objects). In the most difficult regime ($m=95$), the training set
consists of problems that are created using only 5 objects, and the test
problems are created using the remaining 95 objects, whereas in the easiest
regime ($m=0$), both training and test problems are created from the complete
set of 100 objects (though the arrangement of these objects is distinct, such
that the training and test sets are still disjoint). The most difficult
generalization regime poses an especially difficult test of systematic
generalization, as it requires learning of an abstract rule from a very small
set of examples (details shown in Table S1), with little perceptual overlap
between training and test.

As originally proposed [51], the ART dataset involved pre-segmented visual
objects. Here, we investigated a version of this task involving multi-object
visual displays (see Supplementary Section S2 for details). We also applied
random spatial jitter (random translation of up to 5 pixels in any direction)
to each object, as this was previously found to significantly impair
performance in some previous reasoning models [42].

SVRT. The Synthetic Visual Reasoning Test (SVRT), proposed by Fleuret et al.
[13], consists of 23 binary classification tasks, each defined by a particular
configuration of relations. The tasks can be broadly divided into two
categories: those that are based primarily on same/different relations (‘SD’;
tasks 7, 21, 5, 19, 6, 17, 20, 1, 13, 22, 16; example shown in Figure
SLABEL:svrt_sd_example), and those that are primarily based on spatial
relations (‘SR’; tasks 12, 15, 14, 9, 23, 8, 10, 18, 4, 3, 11, 2; example
shown in Figure SLABEL:svrt_sr_example). As with previous work [43], we limit
training to a relatively small number of examples (either 500 or 1000 per
task).

CLEVR-ART. We created a novel dataset based on ART using realistically
rendered 3D shapes from CLEVR [21] (Figure S4). We focused on two specific
visual reasoning tasks: relational-match-to-sample and identity rules.
Problems were formed from objects of three shapes (cube, sphere, and
cylinder), three sizes (small, medium, and large), eight colors (gray, red,
blue, green, brown, purple, cyan, and yellow) and two textures (rubber and
metal). To test systematic generalization, the training set consisted of
problems using only small and medium sized rubber cubes in four colors (cyan,
brown, green, and gray) whereas the test set consisted only of problems using
large sized metal spheres and cylinders in four distinct colors (yellow,
purple, blue, and red). The training and test sets thus involved completely
disjoint perceptual features, similar to the original ART.

###  4.2 Baselines

We compared our model to results from a set of baseline models evaluated by
Vaishnav and Serre [43], including their GAMR architecture, ResNet50 [17], and
a version of ResNet that employed self-attention (Attn-ResNet) [42]. To the
best of our knowledge, these baselines represent the best performing models on
the tasks that we investigated. We also investigated an additional set of
baselines that combined our pretrained slot attention module with various
reasoning architectures, including GAMR [43], ESBN [51], Relation Net (RN)
[35], Transformer [45], Interaction Network (IN) [48], and CoRelNet [22].
These baselines benefited from the same degree of pre-training and object-
centric processing as our proposed model, but employed different reasoning
mechanisms. Finally, we investigated a self-supervised baseline (a Masked
Autoencoder (MAE) [16]) to assess the extent to which a generative objective
might encourage systematic generalization (see Tables S13-S14 for results).
All baselines were evaluated using multi-object displays. See Supplementary
Section S3for more details on baselines.

###  4.3 Experimental Details

####  4.3.1 Pre-training slot attention

We pre-trained slot attention using a slot-based autoencoder framework. For
ART, we generated a dataset of random multi-object displays (see Figure S2),
utilizing a publicly available repository of unicode character
images222https://github.com/bbvanexttechnologies/unicode-images-database. We
eliminated any characters that were identical to, or closely resembled, the
100 characters present in the ART dataset, resulting in a training set of 180k
images, and a validation set of 20k images. For SVRT, we pre-trained two
versions of slot attention on all tasks, one using 500 and the other using
1000 training examples from each of the 23 tasks. For CLEVR-ART, we generated
a dataset of randomly placed 3D shapes from CLEVR, and pre-trained slot
attention using a training set of 90k images, and validation set of 10k
images.

To pre-train slot attention, we used a simple reconstruction objective and a
slot-based decoder. Specifically, we used a spatial broadcast decoder [47],
which generates both a reconstructed image and a mask for each slot. We then
generated a combined reconstruction, normalizing the masks across slots using
a softmax, and using the normalized masks to compute a weighted average of the
slot-specific reconstructions.

After pre-training, the slot attention parameters were frozen for training on
the downstream reasoning tasks. For ART, we found that despite not having been
trained on any of the objects in the dataset, slot attention was nevertheless
able to generate focused attention maps that corresponded closely to the
location of the objects (see Figures S6-S9).

####  4.3.2 Training details and hyperparameters

Images were resized to $128\times 128$, and pixels were normalized to the
range $[0,1]$. For SVRT, we converted the images to grayscale, and also
applied random online augmentations during training, including horizontal and
vertical flips. Tables S2 and S3 describes the hyperparameters for the
convolutional encoder and spatial broadcast decoder respectively. For slot
attention, we used $K=6$ ($K=7$ for CLEVR-ART) slots, $T=3$ attention
iterations per image, and a dimensionality of $D=64$. Training details are
described in Table S4.

Table S5 gives the hyperparameters for the transformer. To solve the SVRT
tasks, and the same/different ART task, we used a CLS token (analogous to CLS
token in [9]) together with a sigmoidal output layer to generate a binary
label, and trained the model with binary cross-entropy loss. To solve the
RMTS, distribution-of-3, and identity rules tasks, we inserted each candidate
answer choice into the problem, and used a CLS token together with a linear
output layer to generate a score. The scores for all answer choices were
passed through a softmax and the model was trained with cross-entropy loss.
For ART and CLEVR-ART, we used a default learning rate of $8e-5$, and a batch
size of 16. The number of training epochs is displayed for each ART task in
Table S6, and for each CLEVR-ART task in Table S10. The training details for
some baseline models were modified as discussed in Section S3, and detailed in
Tables S7-S9. For SVRT, we used a learning rate of $4e-5$, a batch size of 32,
and trained for 2000 epochs on each task. We used the ADAM optimizer [23], and
implementation was done using the Pytorch library [31].

####  4.3.3 Model selection

After pre-training slot attention for a fixed number of epochs (number of
epochs for each dataset listed in Table S4), we selected the version of the
model from the epoch with the lowest validation loss (on the unsupervised pre-
training task) to use in the visual reasoning tasks. We used a validation set
of 20k images for ART (again excluding the objects used in the ART tasks), 4k
images per task for SVRT, and 10k images for CLEVR-ART.

When training on the primary reasoning tasks, for SVRT, we trained for 2000
epochs on each task, and selected the version of the model from the epoch with
the highest accuracy on the validation set. This is consistent with the
approach taken by Vaishnav and Serre [43], who used validation accuracy as a
criterion for early stopping. For ART and CLEVR-ART, we did not perform any
model (or epoch) selection when training on the primary reasoning tasks.

Figure 3: ART results. Results on the Abstract Reasoning Tasks dataset [51].
OCRA outperformed all baselines for all tasks, with especially strong
performance in the most difficult generalization regime ($m=95$). Results
reflect an average across 10 runs for each model $\pm$ standard error. The X
axis denotes the number of objects (out of 100) withheld during training, with
no objects withheld in the leftmost condition ($m=0$), and almost all objects
withheld in the rightmost condition ($m=95$).

##  5 Results

Figure 3 shows the results on the ART dataset (all results are presented in
tabular form in Tables S11-S17). OCRA achieved state-of-the-art results across
all tasks and generalization regimes, outperforming both the previous state-
of-the-art model (GAMR), and other strong baselines such as the slot-
transformer. This disparity was especially striking in the most difficult
generalization regime ($m=95$), in which models were trained on only 5
objects, and tested on a heldout set of 95 objects. Importantly, no component
of OCRA (including both the slot attention and relational reasoning
components) had been exposed to these heldout objects. OCRA outperformed the
next-best baselines in this regime (GAMR and slot-transformer) by as much as
25%, demonstrating a significant degree of systematic generalization. We also
analyzed OCRA’s learned relational embeddings, finding that they captured
relational information even in this out-of-distribution setting (Figure S5).

Many of the baseline models that we evaluated did not contain a relational
bottleneck (Transformer, IN, RN, GAMR, ResNet), limiting their ability to
generalize relational patterns to novel objects (i.e., higher values of $m$).
The comparison with these models thus demonstrates the importance of the
relational bottleneck, which enables systematic generalization of learned
relational patterns. The ESBN and CoRelNet architectures did include a
relational bottleneck, but they were not designed with multi-object inputs in
mind, and therefore performed poorly on these tasks in all regimes (even when
combined with our slot attention module). This is due primarily to the random
permutation of slots in slot attention, which motivated our positional
embedding variable-binding scheme (allowing OCRA to keep track of which
relations correspond to which pairs of objects).

Figure 4: CLEVR-ART. Results for systematic generalization on two CLEVR-ART
tasks. Results reflect test accuracy averaged over 5 trained networks $\pm$
standard error. SVRT. Results on the Synthetic Visual Reasoning Test (SVRT)
[13]. OCRA and GAMR showed comparable overall performance, with OCRA peforming
marginally better on problems involving same / different relations, and GAMR
performing marginally better on problems involving spatial relations.

On SVRT, OCRA showed comparable overall performance with GAMR, displaying
marginally better performance on tasks characterized by same/different
relations, while GAMR performed marginally better on tasks involving spatial
relations (Figure 4, right). OCRA also outperformed GAMR on CLEVR-ART (Figure
4, left), demonstrating that its capacity for systematic generalization can be
extended to problems involving more realistic visual inputs. Overall, the
results on ART, SVRT, and CLEVR-ART demonstrate that OCRA is capable of
solving a diverse range of relational reasoning problems from complex multi-
object displays.

###  5.1 Ablation Study

We performed an ablation study targeting each of OCRA’s major components,
focusing on the most difficult regime of the relational match-to-sample and
identity rules tasks (Table 1). To evaluate the importance of OCRA’s capacity
for object-centric processing, we tested a version of the model without slot
attention (‘w/o Slot Attention’), in which the visual feature map was instead
divided into a $4\times 4$ grid, treating each image patch as though it were
an ‘object’. We also tested a version in which slot attention was only trained
end-to-end on the visual reasoning tasks (‘w/o Pre-training’), to test the
importance of prior exposure to a diverse range of multi-object displays. To
evaluate the importance of OCRA’s relational embedding method, we tested five
different ablation models. In one model (‘w/o Relational Embeddings’), slot
embeddings were passed directly to the transformer. In another model (‘w/o
Factorized Representation’), we did not compute separate feature and position
embeddings for each object, and instead applied the relational operator
directly to the slot embeddings. In a third model (‘w/o Relational
Bottleneck’), relational embeddings were computed simply by applying an MLP to
each pair of slot embeddings (allowing the model to overfit to the perceptual
details of these embeddings), thus removing OCRA’s inductive bias for
relational abstraction. In a fourth model (‘w/o Inner Product’) the dot
product in the relational operator (Equation 3) was replaced with a learned
bottleneck, in which each pair of slot embeddings was passed to a learned
linear layer with a one-dimensional output. This model tested the extent to
which the inherently relational properties of the dot product are important
for relational abstraction (as opposed to resulting merely from compression to
a single dimension). In a fifth model (‘Replace Inner Product w/ Outer
Product’), the dot product was replaced with an outer product (resulting in a
$D\times D$ matrix, which was then flattened and projected via a learned
linear layer to an embedding of size $D$). Finally, to evaluate the importance
of OCRA’s capacity to process higher-order relations, instead of passing
relational embeddings to a transformer, we instead summed all relation
embeddings elementwise and passed the resulting vector to an MLP, as is done
in the Relation Net (‘w/o Transformer’). All ablation models performed
significantly worse than the complete version of OCRA, demonstrating that all
three elements in our model – object-centric processing, relational
abstraction, and higher-order relations – play an essential role in enabling
systematic generalization.

Table 1: Ablation study on $m=95$ generalization regime for two ART tasks. Results reflect test accuracy averaged over 10 trained networks $\pm$ standard error. | RMTS | ID
---|---|---
OCRA (full model) | 85.31$\pm$2.0 | 92.80$\pm$0.3
w/o Slot Attention | 56.58$\pm$3.1 | 84.42$\pm$1.1
w/o Pre-training | 58.69$\pm$3.6 | 53.48$\pm$4.8
w/o Relational Embeddings | 74.79$\pm$1.8 | 81.21$\pm$1.1
w/o Factorized Representation | 50.26$\pm$0.2 | 75.40$\pm$0.6
w/o Relational Bottleneck | 63.62$\pm$1.0 | 76.56$\pm$1.8
w/o Inner Product | 50.70$\pm$0.4 | 48.07$\pm$2.6
Replace Inner Product w/ Outer Product | 62.84$\pm$1.6 | 69.58$\pm$1.1
w/o Transformer | 51.29$\pm$0.2 | 44.18$\pm$0.6

##  6 Limitations and Future Directions

In the present work, we have presented a model that integrates object-centric
visual processing mechanisms (providing the ability to operate over complex
multi-object visual inputs) with a relational bottleneck (providing a strong
inductive bias for learning relational abstractions that enable human-like
systematic generalization of learned abstract rules). Though this is a
promising step forward, and a clear advance relative to previous models of
abstract visual reasoning, it is likely that scaling the present approach to
real-world settings will pose additional challenges. First, real-world images
are especially challenging for object-centric methods due to a relative lack
of consistent segmentation cues. However, there has recently been significant
progress in this direction [36], in particular by utilizing representations
from large-scale self-supervised methods [54, 7], and it should be possible to
integrate these advances with our proposed framework. Second, the current
approach assumes a fixed number of slot representations, which may not be
ideal for modeling real-world scenes with a highly variable number of objects
[55]. Though we did not find that this was an issue in the present work, in
future work it would be desirable to develop a method for dynamically
modifying the number of slots. Third, OCRA’s relational operator is applied to
all possible pairwise object comparisons, an approach that may not be scalable
to scenes that contain many objects. In future work, this may be addressed by
replacing the transformer component of our model with architectures that are
better designed for high-dimensional inputs [18]. Finally, it is unclear how
our proposed approach may fare in settings that involve more complex real-
world relations, and settings that require the discrimination of multiple
relations at once. It may be beneficial in future work to investigate a
‘multi-head’ version of our proposed model (analogous to multi-head
attention), in which multiple distinct relations are processed in parallel. A
major challenge for future work is to develop models that can match the human
capacity for structured visual reasoning in the context of complex, real-world
visual inputs.

##  7 Acknowledgements

Shanka Subhra Mondal was supported by Office of Naval Research grant
N00014-22-1-2002 during the duration of this work. We would like to thank
Princeton Research Computing, especially William G. Wischer and Josko
Plazonic, for their help with scheduling training jobs on the Princeton
University Della cluster.

## References

  * [1] A. Altabaa, T. Webb, J. Cohen, and J. Lafferty.  Abstractors: Transformer modules for symbolic message passing and relational reasoning.  arXiv preprint arXiv:2304.00195, 2023.
  * [2] V. Bapst, A. Sanchez-Gonzalez, C. Doersch, K. Stachenfeld, P. Kohli, P. Battaglia, and J. Hamrick.  Structured agents for physical construction.  In International conference on machine learning, pages 464–474. PMLR, 2019.
  * [3] D. Barrett, F. Hill, A. Santoro, A. Morcos, and T. Lillicrap.  Measuring abstract reasoning in neural networks.  In International conference on machine learning, pages 511–520. PMLR, 2018.
  * [4] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  arXiv preprint arXiv:1806.01261, 2018.
  * [5] L. Bergen, T. O’Donnell, and D. Bahdanau.  Systematic generalization with edge transformers.  Advances in Neural Information Processing Systems, 34:1390–1402, 2021.
  * [6] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner.  Monet: Unsupervised scene decomposition and representation.  arXiv preprint arXiv:1901.11390, 2019.
  * [7] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [8] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078, 2014.
  * [9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [10] D. Ding, F. Hill, A. Santoro, M. Reynolds, and M. Botvinick.  Attention over learned object embeddings enables complex visual reasoning.  Advances in neural information processing systems, 34:9112–9124, 2021.
  * [11] A. Dittadi, S. Papa, M. De Vita, B. Schölkopf, O. Winther, and F. Locatello.  Generalization and robustness implications in object-centric learning.  arXiv preprint arXiv:2107.00637, 2021.
  * [12] M. Engelcke, O. Parker Jones, and I. Posner.  Genesis-v2: Inferring unordered object representations without iterative refinement.  Advances in Neural Information Processing Systems, 34:8085–8094, 2021.
  * [13] F. Fleuret, T. Li, C. Dubout, E. K. Wampler, S. Yantis, and D. Geman.  Comparing machines and humans on a visual categorization test.  Proceedings of the National Academy of Sciences, 108(43):17621–17625, 2011.
  * [14] A. Gopalakrishnan, S. van Steenkiste, and J. Schmidhuber.  Unsupervised object keypoint learning using local spatial predictability.  arXiv preprint arXiv:2011.12930, 2020.
  * [15] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner.  Multi-object representation learning with iterative variational inference.  In International Conference on Machine Learning, pages 2424–2433. PMLR, 2019.
  * [16] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira.  Perceiver: General perception with iterative attention.  In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  * [19] M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu.  Reasoning about physical interactions with object-oriented prediction and planning.  arXiv preprint arXiv:1812.10972, 2018.
  * [20] J. Jiang, F. Deng, G. Singh, and S. Ahn.  Object-centric slot diffusion.  arXiv preprint arXiv:2303.10834, 2023.
  * [21] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * [22] G. Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, and G. Lajoie.  On neural architecture inductive biases for relational tasks.  arXiv preprint arXiv:2206.05056, 2022.
  * [23] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [24] T. Kipf, E. Van der Pol, and M. Welling.  Contrastive learning of structured world models.  arXiv preprint arXiv:1911.12247, 2019.
  * [25] L. Kotovsky and D. Gentner.  Comparison and categorization in the development of relational similarity.  Child Development, 67(6):2797–2822, 1996.
  * [26] B. Lake and M. Baroni.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In International conference on machine learning, pages 2873–2882. PMLR, 2018.
  * [27] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum.  Human-level concept learning through probabilistic program induction.  Science, 350(6266):1332–1338, 2015.
  * [28] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf.  Object-centric learning with slot attention.  Advances in Neural Information Processing Systems, 33:11525–11538, 2020.
  * [29] B. J. Matlen, D. Gentner, and S. L. Franconeri.  Spatial alignment facilitates visual comparison.  Journal of Experimental Psychology: Human Perception and Performance, 46(5):443, 2020.
  * [30] S. S. Mondal, T. W. Webb, and J. D. Cohen.  Learning to reason over visual objects.  In 11th International Conference on Learning Representations, ICLR, 2023.
  * [31] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  2017\.
  * [32] J. C. Raven.  Progressive matrices: A perceptual test of intelligence, individual form.  London: Lewis, 1938.
  * [33] M. Ricci, J. Kim, and T. Serre.  Same-different problems strain convolutional neural networks.  arXiv preprint arXiv:1802.03390, 2018.
  * [34] A. Santoro, R. Faulkner, D. Raposo, J. Rae, M. Chrzanowski, T. Weber, D. Wierstra, O. Vinyals, R. Pascanu, and T. Lillicrap.  Relational recurrent neural networks.  Advances in neural information processing systems, 31, 2018.
  * [35] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap.  A simple neural network module for relational reasoning.  Advances in neural information processing systems, 30, 2017.
  * [36] M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Simon-Gabriel, T. He, Z. Zhang, B. Schölkopf, T. Brox, et al.  Bridging the gap to real-world object-centric learning.  arXiv preprint arXiv:2209.14860, 2022.
  * [37] M. Shanahan, K. Nikiforou, A. Creswell, C. Kaplanis, D. Barrett, and M. Garnelo.  An explicitly relational neural network architecture.  In International Conference on Machine Learning, pages 8593–8603. PMLR, 2020.
  * [38] P. Shaw, J. Uszkoreit, and A. Vaswani.  Self-attention with relative position representations.  arXiv preprint arXiv:1803.02155, 2018.
  * [39] G. Singh, Y.-F. Wu, and S. Ahn.  Simple unsupervised object-centric learning for complex and naturalistic videos.  arXiv preprint arXiv:2205.14065, 2022.
  * [40] A. Stanić and J. Schmidhuber.  R-sqair: relational sequential attend, infer, repeat.  arXiv preprint arXiv:1910.05231, 2019.
  * [41] A. Stanić, S. van Steenkiste, and J. Schmidhuber.  Hierarchical relational inference.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9730–9738, 2021.
  * [42] M. Vaishnav, R. Cadene, A. Alamia, D. Linsley, R. VanRullen, and T. Serre.  Understanding the computational demands underlying visual reasoning.  Neural Computation, 34(5):1075–1099, 2022.
  * [43] M. Vaishnav and T. Serre.  Gamr: A guided attention model for (visual) reasoning.  In 11th International Conference on Learning Representations, ICLR, 2023.
  * [44] S. Van Steenkiste, M. Chang, K. Greff, and J. Schmidhuber.  Relational neural expectation maximization: Unsupervised discovery of objects and their interactions.  arXiv preprint arXiv:1802.10353, 2018.
  * [45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [46] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and S. Levine.  Entity abstraction in visual model-based reinforcement learning.  In Conference on Robot Learning, pages 1439–1456. PMLR, 2020.
  * [47] N. Watters, L. Matthey, C. P. Burgess, and A. Lerchner.  Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes.  arXiv preprint arXiv:1901.07017, 2019.
  * [48] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti.  Visual interaction networks: Learning a physics simulator from video.  Advances in neural information processing systems, 30, 2017.
  * [49] T. Webb, Z. Dulberg, S. Frankland, A. Petrov, R. O’Reilly, and J. Cohen.  Learning representations that support extrapolation.  In International conference on machine learning, pages 10136–10146. PMLR, 2020.
  * [50] T. W. Webb, S. M. Frankland, A. Altabaa, K. Krishnamurthy, D. Campbell, J. Russin, R. O’Reilly, J. Lafferty, and J. D. Cohen.  The relational bottleneck as an inductive bias for efficient abstraction.  arXiv preprint arXiv:2309.06629, 2023.
  * [51] T. W. Webb, I. Sinha, and J. D. Cohen.  Emergent symbols through binding in external memory.  In 9th International Conference on Learning Representations, ICLR, 2021.
  * [52] Z. Wu, N. Dvornik, K. Greff, T. Kipf, and A. Garg.  Slotformer: Unsupervised visual dynamics simulation with object-centric models.  arXiv preprint arXiv:2210.05861, 2022.
  * [53] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, et al.  Relational deep reinforcement learning.  arXiv preprint arXiv:1806.01830, 2018.
  * [54] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong.  ibot: Image bert pre-training with online tokenizer.  arXiv preprint arXiv:2111.07832, 2021.
  * [55] R. S. Zimmermann, S. van Steenkiste, M. S. Sajjadi, T. Kipf, and K. Greff.  Sensitivity of slot-based object-centric models to their number of slots.  arXiv preprint arXiv:2305.18890, 2023.

Supplementary Material

##  S1 Code and Data Availability

All code can be downloaded from https://github.com/Shanka123/OCRA, which also
contains instructions to generate the CLEVR-ART dataset.

##  S2 Datasets

Figure S1: Abstract Reasoning Tasks (ART). Same/different: Two objects are
presented, and the task is to say whether they are the same or different.
Relational match-to-sample: A source pair of objects is presented that either
instantiates a ‘same’ or ‘different’ relation, and the task is to select the
pair of target objects (out of two pairs) that instantiates the same relation.
Problems were presented in a $2\times 2$ array format, with the source pair
presented in the top row, and a target pair presented in the bottom row
(separate images for each target pair, see Figure S7). Distribution-of-3: A
set of three objects is presented in the first row, and an incomplete set is
presented in the second row. The task is to select the missing object from a
set of four choices. Problems were presented in a $2\times 3$ array format,
with one of the answer choices inserted into the bottom right cell (separate
images for each answer choice, see Figure S8). Identity rules: An abstract
pattern is instantiated in the first row (ABA, ABB, or AAA), and the task is
to select the choice that would result in the same relation being instantiated
in the second row. Problems were presented in the same format as the
distribution-of-3 task (Figure S9).

Figure S2: A few examples of the multi-object inputs used to pre-train slot
attention for ART. Table S1: Number of training and test samples for ART
dataset.
Task |  | $m=95$ | $m=85$ | $m=50$ | $m=0$
---|---|---|---|---|---
SD | Training | 40 | 420 | 4900 | 18810
| Test | 10000 | 10000 | 4900 | 990
RMTS | Training | 480 | 10000 | 10000 | 10000
Dist3 | Training | 360 | 10000 | 10000 | 10000
ID | Training | 8640 | 10000 | 10000 | 10000
| Test | 10000 | 10000 | 10000 | 10000

Figure S4: CLEVR-ART. Relational match-to-sample: Example problem involving
‘different’ relation. Correct answer choice (left image) involves ‘different’
relation for both source pair (back row of objects) and target pair (front row
of objects). Incorrect answer choice (right image) involves ‘same’ relation in
target pair. Identity rules: Example problem involving ABA rule. Correct
answer choice (top left image) involves ABA rule in both back row and front
row of objects. Front right object in the other three images (incorrect
choices) violates this rule.

##  S3 Baselines

###  S3.1 GAMR

For the ART and SVRT datasets, we compared to results for the GAMR baseline as
originally reported in [43]. For CLEVR-ART, we used the implementation of GAMR
provided at https://openreview.net/forum?id=iLMgk2IGNyv (Supplementary
Material).

###  S3.2 ResNet and Attn-ResNet

For ART and SVRT, we compared to results from ResNet50 [17] as originally
reported in [43]. For SVRT, we also compared to results from a version of
ResNet that employed self-attention (Attn-ResNet), as reported in [43]. Attn-
ResNet was similar to ResNet50, except that it included a transformer-style
self-attention layer between ResNet blocks (see [43] for details).

###  S3.3 Slot-based reasoning baselines

We investigated a number of baseline models that combined our pre-trained slot
attention module with alternative reasoning architectures. The slot attention
module was the same as used in OCRA. The details of the reasoning
architectures are described in the following sections. To train these
baselines, we used the same settings as used for OCRA (learning rate, batch
size, and number of training epochs), unless otherwise specified.

####  S3.3.1 Slot-GAMR

To combine slot attention with GAMR, we replaced $\boldsymbol{z}_{img}$ in the
original model (originally a flattened feature map from a convolutional
encoder) with the concatenated slot embeddings
$\boldsymbol{slots}\in\mathbb{R}^{K\times D}$. We trained slot-GAMR with a
batch size of 32. The learning rate and number of training epochs for all ART
tasks is given in Tables S8 and S7 respectively. These training details were
based on those used in the original work [43], except that we used a different
learning rate and trained the model for longer in some cases in order to
achieve convergence on the training set.

####  S3.3.2 Slot-ESBN

To combine slot attention with the Emergent Symbol Binding Network (ESBN), the
slot embeddings were passed sequentially to the ESBN, replacing
$\boldsymbol{z}_{t=1\ldots T}$ in the original architecture (one slot
embedding per timestep). The architectural details for ESBN were the same as
reported in [51], except that the output layer from the LSTM controller was
changed based on the task (as described in Section 4.3.2). We used the
implementation provided at https://github.com/taylorwwebb/emergent_symbols. We
trained slot-ESBN with a batch size of 32 and a learning rate of $5e-4$. The
number of training epochs for all ART tasks is given in Table S9. These
training details were based on those used in the original work [51], except
that we trained the model for longer in some cases.

####  S3.3.3 Slot-RN

For the slot-RN baseline, we first passed each pair of slot embeddings through
a shared MLP (referred to as $g_{\theta}$ in the RN framework [35]), with a
hidden layer of size 512, an output layer of size 256, and ReLU nonlinearities
in both layers. The outputs of this MLP were then summed elementwise and
passed to a second MLP ($f_{\phi}$), with a hidden layer of size 256 and ReLU
nonlinearities, and a single output unit (the nonlinearity applied to this
output depended on the task, as described in Section 4.3.2).

####  S3.3.4 Slot-Transformer

For the slot-transformer baseline, we passed the slot embeddings directly to
the transformer used in OCRA. Note that this is identical to the ablation
model referred to as ‘- Relation Embeddings’ in Table 1.

####  S3.3.5 Slot-IN

For the Slot-IN baseline, the slot embeddings were passed to an Interaction
Net (IN), with hyperparameters identical to those described in [48] (the
‘Interaction Net’ described in Section 8.2 of that work). Slot embeddings were
updated for $T=6$ iterations, then summed elementwise and passed through a
final MLP with a hidden layer of size 64 and ReLU nonlinearities, and a single
output unit (with the nonlinearity depending on the task, as described in
Section 4.3.2).

####  S3.3.6 Slot-CoRelNet

For the Slot-CoRelNet baseline, we computed the matrix of all pairwise dot
products between slot embeddings, applied a softmax function across the rows
of this matrix, flattened the matrix and passed it to an MLP with the same
hyperparameters described in [22].

###  S3.4 MAE

We applied the Masked Autoencoder (MAE) model [16] on the identity rules and
distribution-of-3 ART tasks by masking out the final object in each problem
(in the bottom right cell of the input), and training the model to fill in
this patch. To select from the set of multiple choices, we then compared the
model’s generated output with the four answer choices, and selected the choice
with lowest mean-squared error. We used the same hyperparameters described in
[16] for the MAE architecture. We trained the model for 400 epochs with a
batch size of 64, using a learning rate of $2.5e^{-4}$ with warmup for 40
epochs followed by cosine learning rate decay. We did not test this model on
the same/different or RMTS tasks, as it was not clear how to formulate these
tasks in a generative manner.

##  S4 Training details and hyperparameters

Before computing relational embeddings, we applied temporal context
normalization (TCN) [49] to both the the feature embeddings and the position
embeddings. TCN normalizes representations across the temporal dimension, and
has been shown to improve generalization in relational reasoning tasks. When
applying the relational operator $\phi$, we also softmax-normalized the dot
products for all pairwise comparisons.

As shown in Table S4, we pretrained slot attention for SVRT using 500 training
examples for each of the 23 tasks with a learning rate of $4e-4$ for the first
1350 epochs, followed by using a learning rate of $8e-5$ for the next 4620
epochs. When using 1000 training examples for each of the 23 tasks we first
pretrained slot attention with a learning rate of $4e-4$ for the first 1500
epochs, followed by using a learning rate of $8e-5$ for the next 3510 epochs.

Table S2: CNN encoder hyperparameters.
Type | Channels | Activation | Kernel Size | Stride | Padding
---|---|---|---|---|---
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
Position Embedding | - | - | - | - | -
Flatten | - | - | - | - | -
Layer Norm | - | - | - | - | -
1D Conv | 64 | ReLU | 1 | 1 | 0
1D Conv | 64 | - | 1 | 1 | 0

Table S3: Slot decoder hyperparameters.
Type | Channels | Activation | Kernel Size | Stride | Padding
---|---|---|---|---|---
Spatial Broadcast | - | - | - | - | -
Position Embedding | - | - | - | - | -
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 64 | ReLU | $5\times 5$ | 1 | 2
2D Conv | 2 | - | $3\times 3$ | 1 | 1
Table S4: Slot attention pre-training details for all datasets. | ART | SVRT | SVRT | CLEVR-ART
---|---|---|---|---
|  | Dataset Size = 0.5k | Dataset Size = 1k |
Batch size | 32 | 64 | 64 | 32
Learning rate | $8e-5$ |  $4e-4$, $8e-5$ |  $4e-4$, $8e-5$ | $8e-5$
LR warmup steps | 150k | 9k | 18k | 90k
Epochs | 750 | 1350, 4620 | 1500, 3510 | 1000
Table S5: Hyperparameters for Transformer reasoning module. $H$ is the number
of heads, $L$ is the number of layers, $D_{head}$ is the dimensionality of
each head, and $D_{MLP}$ is the dimensionality of the MLP hidden layer.
| ART | SVRT | CLEVR-ART
---|---|---|---
$H$ | 8 | 8 | 8
$L$ | 6 | 24 | 24
$D_{head}$ | 64 | 64 | 64
$D_{MLP}$ | 512 | 512 | 512
Dropout | 0.1 | 0 | 0
Table S6: Default number of training epochs for ART tasks.
Task | $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
SD | 100 | 100 | 100 | 600
RMTS | 100 | 100 | 100 | 400
Dist3 | 100 | 100 | 100 | 400
ID | 100 | 100 | 100 | 100
Table S7: Number of training epochs for slot-GAMR baseline on ART tasks.
Task | $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
SD | 50 | 50 | 100 | 200
RMTS | 100 | 50 | 50 | 300
Dist3 | 100 | 150 | 150 | 300
ID | 100 | 100 | 50 | 100
Table S8: Learning rate for slot-GAMR baseline on ART tasks.
Task | $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
SD | $1e-4$ | $5e-4$ | $5e-4$ | $1e-3$
RMTS | $5e-4$ | $1e-4$ | $5e-4$ | $5e-4$
Dist3 | $5e-4$ | $1e-4$ | $5e-5$ | $5e-4$
ID | $5e-4$ | $5e-4$ | $5e-4$ | $5e-4$

Table S9: Number of training epochs for slot-ESBN baseline on ART tasks.
Task | $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
SD | 150 | 150 | 150 | 200
RMTS | 150 | 150 | 150 | 200
Dist3 | 150 | 150 | 150 | 200
ID | 150 | 150 | 150 | 150
Table S10: Number of training epochs for CLEVR-ART tasks. OCRA was trained for
longer on the identity rules (ID) task in order to achieve convergence on the
training set. 50 epochs was sufficient to achieve convergence on both tasks
for GAMR, and on RMTS for OCRA.
Task | OCRA | GAMR
---|---|---
RMTS | 50 | 50
ID | 200 | 50

##  S5 Results

Table S11: Results for ART same/different task.
| $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
OCRA (ours) | 97.31$\pm$1.5 | 95.98$\pm$1.1 | 96.48$\pm$0.3 | 87.95$\pm$1.3
Slot-Transformer | 98.47$\pm$0.6 | 95.95$\pm$0.5 |  87.53$\pm$1.1 |  68.46$\pm$2.0
Slot-IN |  94.59$\pm$1.4 |  92.43$\pm$0.8 |  68.4$\pm$4.3 |  59.23$\pm$2.3
Slot-RN |  93.97$\pm$2.8 |  94.23$\pm$0.7 |  83.66$\pm$1.2 |  77.26$\pm$1.9
Slot-GAMR |  61.43$\pm$2.7 |  67.35$\pm$1.1 |  63.97$\pm$0.5 |  62.98$\pm$1.4
Slot-ESBN |  65.42$\pm$6.2 |  54.1$\pm$3.7 |  49.79$\pm$0.1 |  50.02$\pm$0.2
Slot-CoRelNet |  58.24$\pm$2.1 |  55.91$\pm$0.3 |  50.70$\pm$0.2 |  50.50$\pm$0.2
GAMR | 97.28$\pm$0.5 | 94.4$\pm$0.3 |  87.88$\pm$1.3 |  83.49$\pm$1.4
ResNet |  95.65$\pm$0.7 |  92.23$\pm$0.6 |  82.83$\pm$1.4 |  66.6$\pm$1.5
Table S12: Results for ART relational-match-to-sample task.
| $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
OCRA (ours) | 99.91$\pm$0.0 | 99.25$\pm$0.1 | 94.43$\pm$0.7 | 85.31$\pm$2.0
Slot-Transformer | 99.73$\pm$0.0 | 99.12$\pm$0.1 | 96.77$\pm$0.3 |  73.99$\pm$3.0
Slot-IN |  93.42$\pm$0.7 |  87.42$\pm$0.7 |  86.71$\pm$1.8 |  56.93$\pm$0.8
Slot-RN |  91.31$\pm$0.3 |  87.1$\pm$0.9 |  68.4$\pm$1.3 |  61.62$\pm$1.1
Slot-GAMR |  80.92$\pm$6.9 |  93.43$\pm$0.4 |  61.47$\pm$0.8 |  59.55$\pm$2.7
Slot-ESBN |  50.63$\pm$0.4 |  50.3$\pm$0.4 |  57.12$\pm$3.0 |  49.99$\pm$0.2
Slot-CoRelNet |  50.18$\pm$0.1 |  50.30$\pm$0.2 |  49.84$\pm$0.2 |  49.82$\pm$0.2
GAMR | 98.12$\pm$0.4 |  91.98$\pm$0.8 |  89.81$\pm$0.5 |  72.2$\pm$3.0
ResNet |  53.77$\pm$4.0 |  69.43$\pm$6.5 |  82.18$\pm$1.0 |  49.89$\pm$0.2

Table S13: Results for ART distribution-of-3 task.
| $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
OCRA (ours) | 98.86$\pm$0.2 | 97.87$\pm$0.3 | 96.09$\pm$0.4 | 86.42$\pm$1.3
Slot-Transformer | 99.49$\pm$0.0 | 99.08$\pm$0.1 | 95.82$\pm$0.5 |  60.61$\pm$1.9
Slot-IN |  97.12$\pm$0.2 |  96.22$\pm$0.3 |  90.51$\pm$2.7 |  49.48$\pm$1.8
Slot-RN |  97.8$\pm$0.1 |  96.82$\pm$0.1 |  91.14$\pm$0.5 |  52.1$\pm$0.7
Slot-GAMR |  73.74$\pm$0.7 |  68.91$\pm$0.6 |  58.3$\pm$0.5 |  32.77$\pm$1.0
Slot-ESBN |  39.79$\pm$1.2 |  38.8$\pm$2.4 |  33.86$\pm$2.5 |  25.56$\pm$0.1
Slot-CoRelNet |  28.98$\pm$0.1 |  28.78$\pm$0.1 |  28.58$\pm$0.3 |  26.80$\pm$0.8
GAMR |  91.98$\pm$0.5 |  87.3$\pm$0.4 |  80.57$\pm$1.4 |  68.62$\pm$1.8
ResNet |  88.61$\pm$0.3 |  84.32$\pm$0.7 |  74.82$\pm$1.3 |  50.07$\pm$1.3
MAE | 99.99$\pm$0.0 |  56.47$\pm$1.1 |  40.90$\pm$1.2 |  28.85$\pm$0.9
Table S14: Results for ART identity rules task.
| $m=0$ | $m=50$ | $m=85$ | $m=95$
---|---|---|---|---
OCRA (ours) | 99.01$\pm$0.0 | 98.01$\pm$0.1 | 96.67$\pm$0.2 | 92.8$\pm$0.3
Slot-Transformer | 99.44$\pm$0.0 | 99.06$\pm$0.1 | 97.15$\pm$0.3 |  78.32$\pm$1.8
Slot-IN |  97.62$\pm$0.5 |  96.72$\pm$0.6 |  94.78$\pm$0.7 |  72.82$\pm$1.6
Slot-RN |  93.44$\pm$0.2 |  91.1$\pm$0.3 |  86.73$\pm$0.3 |  65.96$\pm$1.1
Slot-GAMR |  96.11$\pm$0.1 |  94.8$\pm$0.2 |  76.52$\pm$3.0 |  61.92$\pm$0.9
Slot-ESBN |  54.83$\pm$0.8 |  55.7$\pm$0.7 |  49.52$\pm$3.2 |  50.33$\pm$2.8
Slot-CoRelNet |  42.28$\pm$4.7 |  39.84$\pm$4.4 |  35.19$\pm$4.0 |  43.50$\pm$5.2
GAMR |  96.89$\pm$0.5 |  91.9$\pm$0.7 |  85.44$\pm$1.2 |  66.23$\pm$4.8
ResNet |  93.27$\pm$0.6 |  84.97$\pm$0.6 |  77.94$\pm$1.1 |  54.84$\pm$2.4
MAE |  66.55$\pm$0.2 |  44.96$\pm$0.6 |  38.39$\pm$0.9 |  31.56$\pm$1.0
Table S15: Results for systematic generalization on two CLEVR-ART tasks. Results reflect test accuracy averaged over 5 trained networks ($\pm$ standard error). | RMTS | ID
---|---|---
GAMR | 70.40$\pm$5.8 | 74.15$\pm$4.0
Slot-Transformer | 87.54$\pm$0.7 | 78.81$\pm$1.6
Slot-IN | 66.72$\pm$3.7 | 67.22$\pm$1.7
Slot-RN | 64.79$\pm$0.5 | 60.27$\pm$0.6
Slot-GAMR | 52.56$\pm$0.5 | 39.83$\pm$0.9
Slot-ESBN | 62.53$\pm$0.1 | 28.87$\pm$0.7
Slot-CoRelNet | 49.87$\pm$0.2 | 24.80$\pm$0.3
OCRA | 93.34$\pm$1.0 | 77.06$\pm$0.7

Table S16: Results for SVRT same/different tasks.
| Dataset Size =1k | Dataset Size =0.5k
---|---|---
OCRA (ours) | 90.30$\pm$4.1 | 79.89$\pm$4.5
Slot-Transformer | 89.85$\pm$4.2 | 76.54$\pm$5.1
Slot-IN |  74.99$\pm$4.9 |  68.23$\pm$4.8
Slot-RN |  81.79$\pm$4.4 |  71.48$\pm$4.8
Slot-GAMR |  66.87$\pm$3.2 |  63.06$\pm$3.7
Slot-ESBN |  51.67$\pm$1.1 |  53.83$\pm$1.1
Slot-CoRelNet |  57.13$\pm$2.6 |  52.95$\pm$1.4
GAMR |  82.05$\pm$4.4 | 76.80$\pm$4.9
Attn-ResNet |  68.83$\pm$4.4 |  62.30$\pm$3.5
ResNet |  56.88$\pm$2.5 |  54.97$\pm$2.2
Table S17: Results for SVRT spatial-relations tasks.
| Dataset Size =1k | Dataset Size =0.5k
---|---|---
OCRA (ours) | $95.02\pm 2.4$ | $89.25\pm 2.5$
Slot-Transformer | 97.86$\pm$0.9 |  94.06$\pm 1.6$
Slot-IN |  94.86$\pm$1.4 |  90.23$\pm$2.0
Slot-RN |  96.20$\pm$1.4 |  91.73$\pm$1.8
Slot-GAMR |  86.99$\pm$2.2 |  84.90$\pm$2.4
Slot-ESBN |  62.69$\pm$2.4 |  61.30$\pm$2.3
Slot-CoRelNet |  74.59$\pm$3.7 |  60.95$\pm$3.7
GAMR | 98.74$\pm$0.3 | 97.40$\pm$0.7
Attn-ResNet |  97.66$\pm$0.7 |  94.80$\pm$1.4
ResNet |  94.87$\pm$1.6 |  85.18$\pm$4.3

Figure S5: Visualization of learned relation embeddings. OCRA’s relation
embeddings (projected on to top 2 principal components) for test set in $m=95$
regime of same/different ART task. Despite completely novel objects, relation
embeddings display separation based on relational category.
Figure S6: Slot-specific attention maps applied to input image for
same/different task.
Figure S7: Slot-specific attention maps applied to input image for relational
match-to-sample task.
Figure S8: Slot-specific attention maps applied to input image for
distribution-of-three task.
Figure S9: Slot-specific attention maps applied to input image for identity
rules task.


11institutetext: Departament de Física. Universitat Politècnica de Catalunya
(Spain) 22institutetext: Scientific Computing Center (sciCORE). Universität
Basel (Switzerland) 33institutetext: Institut d’Estudis Espacials de Catalunya

# A moderately-sized nuclear network to assist multi-D hydrodynamic
simulations of supernova explosions

Axel Sanz 11<EMAIL_ADDRESS>Rubén Cabezón 22
<EMAIL_ADDRESS>Domingo García-Senz 1133<EMAIL_ADDRESS>
###### Abstract

A key ingredient in any numerical study of supernova explosions is the nuclear
network routine that is coupled with the hydrodynamic simulation code. When
these studies are performed in more than one dimension, the size of the
network is severely limited by computational issues. In this work, we propose
a nuclear network (net87) which is close to one hundred nuclei and could be
appropriate to simulate supernova explosions in multidimensional studies. One
relevant feature is that electron and positron captures on free protons and
neutrons have been incorporated to the network. Such addition allows for a
better track of both, the neutronized species and the gas pressure. A second
important feature is that the reactions are implicitly coupled with the
temperature, which enhances the stability in the nuclear statistical
equilibrium (NSE) regime. Here we analyze the performance of net87 in light of
both: the computational overhead of the algorithm and the outcome in terms of
the released nuclear energy and produced yields in typical Type Ia Supernova
conditions.

##  1 Introduction

The disrupting mechanism in Type Ia Supernova (SN Ia) explosions is of
thermonuclear origin and, therefore, a strong feedback between the
hydrodynamics and the released nuclear energy is expected Hillebrandt &
Niemeyer (2000). Any hydrodynamic code aiming to simulate these explosions in
more than one dimension has to incorporate good enough nuclear routines,
although not too time-consuming in computing terms.

Pioneering multidimensional simulations of SN Ia incorporated around 10 nuclei
with the aim of having, at best, a reasonable depiction of the released
nuclear energy. For example, Benz et al. Benz et al. (1989) implemented an
$\alpha$-network to keep track of the nuclear evolution during the collision
of two white dwarfs. Timmes et al. Timmes et al. (2000) analyzed the
performance of two networks, with 7 and 14 species, concluding that they are
able to account for the nuclear generation rate within $20\%$ precision. The
use of these small networks has been commonplace in multi-D simulations of
supernova explosions. See Reinecke et al. (1999); García-Senz et al. (1999);
Plewa et al. (2004), as an example of works connected with different SN Ia
explosion scenarios. At present, multidimensional simulations incorporate
$\simeq 50$ nuclei, being able to reproduce the released nuclear energy within
a narrow deviation, $<5\%$ with respect a larger network Gronow et al. (2021).

In this work, we describe a network of 87 species, net87, which can be used in
explosive astrophysical scenarios. In particular, we applied it to assist with
the explosion of a massive white dwarf (WD) made of 12C$+^{16}$O with central
density $\rho_{c}\simeq 2\times 10^{9}$ g cm-3 in the so-called Chandrasekhar-
mass explosion models. It is also adequate to track the detonation of a tiny
helium shell at densities $\simeq 5\times 10^{5}$ g cm-3 located on top of a
moderately massive WD, which is a necessary condition to study the
SubChandrasekhar-mass route to SN Ia.

Our proposal has two important features. First, the molar fractions of the
species, $Y_{i}$, and the temperature, $T$, are found jointly, after
implicitly solving the system of nuclear reactions coupled with the energy
equation. It is well known that the implicit coupling of $\\{Y_{i},T\\}$ leads
to a much stable behavior around the NSE regime Mueller (1986); Cabezón et al.
(2004) and recent studies have identified the implicit coupling between the
released nuclear energy and the nuclear reaction rates as a key point in
simulating high-density combustion (Zingale et al., 2021). The second novelty
of this work is that our moderately-sized network includes the capture
reactions $e^{-}+p\rightarrow n+\nu_{e}$ and $e^{+}+n\rightarrow p+\bar{\nu}$.
As far as we know, this is the first time that reactions of this kind are
taken into account in a reduced network routine belonging to a multi-D
hydrodynamic code dealing with SN Ia explosions. As we show below, the
benefits of including these reactions -namely a more accurate depiction of the
electronic pressure and neutronized yields- come at no relevant computational
cost.

##  2 Calculations and results

net87 adds a cluster of proton and neutron reactions in between each node of a
standard $\alpha-$network from 20Ne up to 60Zn (see Fig. 1). The variation of
the molar fractions is integrated jointly with the temperature equation with
an implicit Newton-Raphson scheme. We also included $e^{-}$ and $e^{+}$
captures on free protons and neutrons. They have well-known rates Fuller et
al. (1980) and may account for up to $60\%$ of bulk captures for
Chandrasekhar-mass SN explosions.

Figure 2 shows the results of a 12C$+^{16}$O burning test with initial
temperature $T_{9}=1$ and density $\rho_{9}=2$. An artificial decrease of
density, simulating an adiabatic expansion, was switched on around iteration
400, well after NSE is established, to reach the freeze-out. Note the decrease
in pressure and temperature when the endoenergetic e--captures become
relevant, around iteration 300. Such reduction in pressure will have dynamical
consequences in multi-D simulations of SN Ia. Additionally, the NSE plateau is
well reproduced, with the abundances showing no oscillations.

Figure 1: Schematics of the net87 network, showing the $\alpha$, $p$, and $n$
reactions in the first and last cluster nodes (in purple) between
$\alpha$-chain elements (in black). Three heavy-ion reactions are also
included as they are the main seeds for $\alpha$ particles. Finally, the
$e^{-}$ and $e^{+}$ captures are depicted on the top-left.

Table 1 shows the computational performance of net87. The routine scales
approximately as $N\log N$ with the number of species as going from net14
(with 14 species) to net87 rises the computing time in a factor $\simeq 15$.
The overload of including $e^{-}$ and $e^{+}$ captures is negligible.

Table 1:  Wall-clock time, average number of Newton-Raphson iterations per global iteration, total number of global iterations, and average wall-clock time per single iteration for the 12C$+^{16}$O test and for each nuclear network (net86 is net87 without $e^{-}$, $e^{+}$-captures) Network | $\overline{t}$ (s) | $\overline{\textrm{NR It.}}$ | It. |  $\overline{t}$/It.
---|---|---|---|---
($\times 10^{-4}$ s)
net14 | $0.152\pm 0.009$ | $4.23$ | $1792$ | $0.20$
net86 | $1.92\pm 0.13$ | $4.89$ | $1265$ | $3.10$
net87 | $1.78\pm 0.08$ | $4.87$ | $1164$ | $3.08$

Figure 2: Evolution of the most relevant quantities in the 12C$+^{16}$O test.
Top-left: nuclear abundances for net87; top-right: temperature, density, and
pressure for net87 and net86; bottom: nuclear energy generation rate for
net87, net86 and net14.

## References

  * Hillebrandt & Niemeyer (2000) Hillebrandt, W. & Niemeyer, J. C., ARA$\&$A 38, 191 (2000)
  * Benz et al. (1989) Benz, W., Hills, J. G., & Thielemann, F. K., ApJ 342, 986 (1989)
  * Timmes et al. (2000) Timmes, F. X., Hoffman, R. D., & Woosley, S. E., ApJS 129, 377 (2000)
  * Reinecke et al. (1999) Reinecke, M., Hillebrandt, W., & Niemeyer, J. C., A$\&$A 347, 739 (1999)
  * García-Senz et al. (1999) García-Senz, D., Bravo, E., & Woosley, S. E., A$\&$A 349, 177 (1999)
  * Plewa et al. (2004) Plewa, T., Calder, A. C., & Lamb, D. Q., ApJL 612, L37 (2004)
  * Gronow et al. (2021) Gronow, S., Collins, C. E., Sim, S. A., & Röpke, F. K., A$\&$A 649, A155 (2021)
  * Mueller (1986) Mueller, E., A$\&$A, 162, 103 (1986)
  * Cabezón et al. (2004) Cabezón, R. M., García-Senz, D., & Bravo, E., ApJS 151, 345 (2004)
  * Zingale et al. (2021) Zingale, M., Katz, M. P., Willcox, D. E., & Harpole, A., Research Notes of the American Astronomical Society 5, 71 (2021)
  * Fuller et al. (1980) Fuller, G. M., Fowler, W. A., & Newman, M. J., ApJS 42, 447 (1980)