# How Modular Should Neural Module Networks Be
for Systematic Generalization?

Vanessa D’Amario 1, 3 &Tomotake Sasaki 2, 3 &Xavier Boix 1, 3 1 Massachusetts
Institute of Technology, USA 2 Fujitsu Limited, Japan
3 Center for Brains, Minds and Machines, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Neural Module Networks (NMNs) aim at Visual Question Answering (VQA) via
composition of modules that tackle a sub-task. NMNs are a promising strategy
to achieve systematic generalization, ie. overcoming biasing factors in the
training distribution. However, the aspects of NMNs that facilitate systematic
generalization are not fully understood. In this paper, we demonstrate that
the degree of modularity of the NMN have large influence on systematic
generalization. In a series of experiments on three VQA datasets (VQA-MNIST,
SQOOP, and CLEVR-CoGenT), our results reveal that tuning the degree of
modularity, especially at the image encoder stage, reaches substantially
higher systematic generalization. These findings lead to new NMN architectures
that outperform previous ones in terms of systematic generalization.

##  1 Introduction

The combinatorial nature of vision [1] is an open challenge for learning
machines tackling Visual Question Answering (VQA) [2, 3, 4, 5, 6, 7]. The
amount of possible combinations of object categories, attributes, relations,
context and visual tasks is exponentially large, and any training set contains
a limited amount of such combinations. The ability to generalize to
combinations not included in the training set, ie. the so-called _systematic
generalization_ [8, 9, 10], is a hallmark of intelligence.

Recent research has shown that Neural Module Networks (NMNs) are a promising
approach for systematic generalization in VQA [11, 5]. NMNs infer a program
layout containing a sequence of sub-tasks, where each sub-task is associated
to a module consisting of a neural network [12, 13].

The aspects that facilitate systematic generalization in NMNs are not fully
understood. Recent works demonstrate that modular approaches are helpful for
object recognition tasks with novel combinations of category and viewpoints
[14] and also novel combinations of visual attributes [15]. Bahdanau et al.
[11] have demonstrated the importance of using an appropriate program layout
for systematic generalization, and Bahdanau et al. [5] in another paper have
shown that the module’s network architecture also impacts systematic
generalization. These results highlight that using modules to tackle sub-tasks
and using compositions of these modules facilitates systematic generalization.
However, there is a central question at the heart of modular approaches that
remains largely unaddressed: what sub-tasks should be tackled by a module?
Namely, is it best for systematic generalization to use a large degree of
modularity such that modules tackle very specific sub-tasks, or is it best to
use a smaller degree of modularity?

In this paper, we investigate what sub-tasks a module should perform in order
to facilitate systematic generalization in VQA. We analyse different degrees
of modularity as in the example of Figure 1, where NMNs could have three
different degrees of modularity: (i) a unique module for tackling all sub-
tasks, (ii) modules tackling groups of sub-tasks, or (iii) many modules each
one for tackling sub-tasks at the highest degree of modularity. Also, we
investigate the degree of modularity at all the stages of the network, as NMNs
have a structure divided in three stages: an _image encoder_ for the
extraction of visual features, a set of _intermediate modules_ arranged based
on the program layout, and a _classifier_ that provides the output answer.

We examined the degree of modularity in three families of VQA datasets: VQA-
MNIST, the SQOOP dataset [11], and the Compositional Generalization Test
(CoGenT) of CLEVR [2]. Our results reveal two main findings that are
consistent across the tested datasets: (i) an intermediate degree of
modularity by grouping sub-tasks leads to much higher systematic
generalization than for the NMNs’ modules introduced in previous works, and
(ii) modularity is mostly effective when is defined at the image encoder
stage, which is rarely done in the literature. Furthermore, we show that these
findings are directly applicable to improve the systematic generalization of
state-of-the-art NMNs.

##  2 Libraries of Modules

NMNs consist of three components: a program generator, a library of modules
and an execution engine. The program generator translates VQA questions into
program layouts, consisting of compositions of sub-tasks. The library of
modules contains neural networks that perform sub-tasks, and these modules are
used by the execution engine to put the program layout into action.

---
(a)
(b)
(c)
Figure 1: _Libraries of modules._ (a) Example of five libraries with different
degrees of modularity (_all_ , _group_ , and _sub-task_) analyzed in this
paper. Some modules have additional input arguments to indicate specifics of
the sub-task. (b) Intermediate modules from libraries with _all_ , _group_ ,
and _sub-task_ degrees of modularity, for the VQA-MNIST dataset. (c)
Composition of modules leveraging on the libraries in (a), given the question
_“Is the green object left of ‘2’?”_ and an image containing a pair of objects
(the former identified by its color, the latter by its category).

Given that the use of a correct program layout plays a fundamental role to
achieve systematic generalization with NMNs [13, 11], we follow the approach
in previous works that fix the program layout to the ground-truth program,
such that it does not interfere in the analysis of the library [11, 5]. Thus,
we follow a reductionist approach, which facilitates gaining an understanding
of the crucial aspects of NMNs in systematic generalization. As we gain such
understanding, we will be in a position to study the interactions between
these aspects in future works.

Here, we aim at understanding whether the choice of a library with a specific
degree of modularity facilitates systematic generalization in NMNs. Figure 1a
shows examples of several libraries of modules (one library per column),
characterized by different degrees of modularity at different stages of the
network. Each row identifies a stage in the network, while the use of single
or multiple modules identifies the degree of modularity at each stage. In
particular, a library with a single module shared across all sub-tasks is
called _all_. A library with modules tackling groups of sub-tasks is called
_group_. Finally, a library with modules that tackle each single sub-task is
called _sub-task_. These different ways of defining the degree of modularity
in the library are detailed in Figure 1b.

Note that in the intermediate stage, it is sometimes necessary to use modules
that operate differently depending on an input argument related to the sub-
task to be performed by the module. For example, a module whose sub-task is to
determine whether an object is of a specific color or not, could be
implemented by a module specialized to detect colors with an input argument
that indicates the color to be detected. It could also be implemented by a
module specialized to detect a specific color, which would not require an
additional input argument.

In Figure 1c, we provide an example of usage of the libraries given the
question _“Is the green object left of ‘2’?”_. The ground-truth program layout
is a tree structure with two leaves and a root that identify respectively the
pair of objects and the spatial relation. In the figure we depict how the same
program layout can be implemented with libraries with different degrees of
modularity. Note that the degree of modularity does not change the final
number of modules in the program layout, but the number of modules in the
library.

In the following, we first introduce the definitions of the modules in the
library for the image encoder and the classifier stages, and then we introduce
existing implementations of intermediate modules and propose a new one.

###  2.1 Libraries of Modules at the Image Encoder and Classifier Stages

The image encoder is usually a single convolutional neural network module
common to all sub-tasks (first two columns of Figure 1a). We analyze image
encoders defined per group of sub-tasks, as in the third and forth columns in
Figure 1a. The library with maximum degree of modularity at the image encoder
stage corresponds to having an image encoder per sub-task of the VQA problem,
as depicted in the last column of Figure 1a.

For a VQA task with binary answers, libraries with different degrees of
modularity can be equally defined on the classifier.

###  2.2 Libraries of Modules at the Intermediate Modules Stage

The architecture of the intermediate modules varies across the literature. We
consider two architectures which previous works [11, 5] have focused on: the
_Find_ [16] and the _Residual_ [13] modules. Let $f(k,s_{x},s_{y})$ be the
function that represents an intermediate module, where each sub-task is
denoted by an index $k$ that represents the input argument of the module, and
($s_{x}$, $s_{y}$) are the data inputs to the module, which can be the
representation of the input image or the output of the precedent intermediate
modules according to the program layout. If a module requires a single input
for the given sub-task, $s_{y}$ is the output of the image encoder given a
null input image.

In the following, we show that the _Find module_ can be interpreted as a
library with a _single intermediate module_ shared across all sub-tasks, with
the neural representation modulated by an embedding related to the sub-task.
Then, we show that the _Residual module_ corresponds to a library with maximum
degree of modularity, in which each sub-task corresponds a single intermediate
module. Finally, we introduce a new module based on these two, which has an
intermediate degree of modularity.

#### A single intermediate module (aka. Find module [16]).

The definition is the following:

$\displaystyle\gamma_{k}=\text{Embedding}(k),$ (1) $\displaystyle
f(k,s_{x},s_{y})=\text{ReLU}(W_{1}*(\gamma_{k}\odot\text{ReLU}(W_{2}*[s_{x};s_{y}]+b_{2}))+b_{1}),$
(2)

where $[s_{x};s_{y}]$ is the concatenation of the two inputs and
$[W_{1};b_{1};W_{2};b_{2}]$ are the weights and bias terms of the
convolutional layer. These weights and bias terms are the same across all sub-
tasks, ie. all the sub-tasks are tackled with the same module. The module
performs a sub-task by “modulating” the neural activity via the element-wise
product involving $\gamma_{k}$. Figure 1b depicts the Find module at the first
column.

#### One module per sub-task (aka. Residual module [13]).

This represents a library with a much finer degree of modularity than the
previous one. For each sub-task $k$, we have an independent set of
convolutional weights indexed by $k$, ie.
$[W_{1}^{k};b_{1}^{k};W_{2}^{k};b_{2}^{k};W_{3}^{k};b_{3}^{k}]$, as in the
following:

$\displaystyle\tilde{s}_{k}=\text{ReLU}(W_{3}^{k}*[s_{x};s_{y}]+b_{3}^{k}),$
(3) $\displaystyle
f(k,s_{x},s_{y})=\text{ReLU}(\tilde{s}_{k}+W_{1}^{k}*\text{ReLU}(W_{2}^{k}*\tilde{s}_{k}+b_{2}^{k})+b_{1}^{k}).$
(4)

Note that here the mechanism to tackle a sub-task is to use an independent set
of weights per sub-task, as depicted in Figure 1b, at the third column.

#### One module per group of sub-tasks.

We introduce a new library of modules that has a degree of modularity halfway
between the two presented above (depicted in Figure 1b, second column). Each
modules tackles a group of sub-tasks via a Find module. We use
$[W^{g}_{1};b^{g}_{1};W^{g}_{2};b^{g}_{2}]$ to denote the weights and bias
terms of a module for the group of sub-tasks indexed by $g$. Let
$g=\mbox{Group}(k)$ be a mapping from the sub-task $k$ to the index of the
group, $g$. Thus, this library of modules is defined as follows:

$\displaystyle g=\mbox{Group}(k),\;\;\gamma_{k}=\text{Embedding}(k),$ (5)
$\displaystyle
f(k,s_{x},s_{y})=\text{ReLU}(W^{g}_{1}*(\gamma_{k}\odot\text{ReLU}(W^{g}_{2}*[s_{x},s_{y}]+b^{g}_{2}))+b^{g}_{1}).$
(6)

Note that this library of modules uses both Find and Residual architectures to
tackle a sub-task—the division of sub-tasks in groups of separated weights
reflects the design of the Residual architecture, while the modulation
mechanism that allows the modules to adjust their representation among sub-
tasks in the same group reflects the design of the Find architecture.

##  3 Datasets for Analysing Systematic Generalization in VQA

In this section, we introduce the VQA-MNIST and the SQOOP datasets to study
the effect of libraries with different degrees of modularity on systematic
generalization.

###  3.1 VQA-MNIST: limited combinations of visual attributes

We introduce VQA-MNIST, a family of four VQA datasets of which two of them are
related to attribute extraction, while the other two deal with comparisons of
attributes and spatial positions. All tasks require a binary answer (‘yes’ or
‘no’). Each object is an MNIST digit [17] with multiple attributes: category,
color, size and illumination condition among respectively ten, five, three,
and three different options. This leads to a total amount of attribute
instances equal to $21$, and a total amount of combinations of attribute
instances equal to $450$. See Appendix A.1 for more details.

---
Figure 2: _VQA datasets to evaluate systematic generalization._ From top to
bottom: Visual input, question and corresponding program layout. The VQA-MNIST
contains objects that are combinations of four attributes (category, color,
size and illumination condition). The tasks consist on attribute extraction
(from a single object and from multiple objects), and attribute comparisons
(between separated object pairs and spatial locations). The training
distribution has a limited amount of attribute combinations. The SQOOP dataset
contains objects at different positions and the task consists on comparing the
position of two given objects. The training distribution has a limited number
of co-occurring objects in the image. The original SQOOP dataset contains five
objects per image [11], and we further limit the training distribution by only
allowing two objects per image.

To generate a limited set of combinations of attributes, we fix a parameter
$r$ and we randomly extract combinations among all the $450$ possibilities,
with the requirement that each of the $21$ attribute instances appears at
least $r$ times (see Appendix A.2 for details). In order to study the effect
of training data diversity, we compare NMNs trained with datasets with
different number of attribute combinations and for a fair comparison, we
evaluate them under the same testing conditions, as described next. We define
_data-run_ as six different training datasets originated from training
combinations obtained with six different $r$’s ($[1,2,5,8,10,20]$), such that
the set of training combinations extracted for smaller values of $r$ is
included in the training combinations for larger values of $r$. We evaluate
systematic generalization for all datasets in the data-run on the same images
that come from novel combinations, obtained by fixing $r=5$. We generate the
training and test examples as described in Appendix A.3. The amount of
training examples for each dataset in the data-run is fixed to $210$K for
training, and $42$K for testing systematic generalization. In this way, all
NMNs in the data-run are trained with the same amount of data, ie. we evaluate
different amounts of training data diversity for the same number of training
images.

In Figure 2, we depict the four VQA tasks divided into _attribute extraction_
and _attribute comparison_ , which are summarized in the following:

  * •

_Attribute extraction._ Starting from the left of Figure 2, we show two
datasets, _attribute extraction from single object_ and _attribute extraction
from multiple objects_. In the former case, the image contains a single object
(image size is $28\times 28$ pixels), in the latter, it always contains a pair
of objects (image size is $64\times 64$ pixels). The second object plays the
role of a confounding factor. In both datasets, the questions inquire about
one of the $21$ possible attribute instances. The ground-truth program layout
corresponds to a single sub-task, which is identified by the attribute
instance in the question. Thus, the program layout in the intermediate module
corresponds to a single module identifying the sub-task (the sub-tasks in
Figure 2 are identified by _‘7’_ and _green_).

  * •

_Attribute comparison._ At the center of Figure 2 we show two datasets for
attribute comparison. These have a ground-truth program layout that is a tree
structure. The first dataset consists on _attribute comparison between pairs
of separated objects_ , in which comparisons are done between a pair of
objects contained in separated images (of size $28\times 28$ pixels). The
comparison are with respect to an attribute of the dataset. The two leaves in
the program layout extract the attribute and the root tackles the comparison
between them. The second dataset consists on a _comparison between spatial
positions_ , as shown in Figure 2 (image size $64\times 64$ pixels). The
leaves of the program layout identify the objects in the image, and the root
tackles the spatial comparison (ie. above, below, left of, or right of).

###  3.2 SQOOP: limited amount of co-occurrence of objects in the images

The SQOOP dataset tackles the problem of spatial relations between two objects
[11]. Namely, the questions consist on comparing the position of two objects,
_e.g., Is D left of G?_ The dataset contains four spatial relations (above,
below, left of, or right of) on thirty-six possible objects. All objects can
appear in the image, but specific pairs of objects are not compared during
training. By limiting the possible pairs of objects compared during training,
systematic generalization is measured as the ability to compare novel pairs of
objects.

In the original SQOOP dataset [11], each image contains five objects, as in
the left example of the SQOOP block in Figure 2. We increase the bias of the
SQOOP dataset, to make it more challenging for NMNs. To do so, we reduce the
objects in the image to the pair in the question. We generate a dataset with
the smallest amount of objects co-occurring with another object in an image,
such that each object only co-occurs with another object. Given a generic
training question _e.g., “Is D left of G?”_ , all the training images related
to this question contain the _D_ and _G_ objects only. This procedure
substantially reduces the training data diversity of the original SQOOP
datasets with five objects per image.

##  4 Results

In this section, we report the systematic generalization accuracy of NMNs with
different libraries of modules on the VQA-MNIST and the SQOOP datasets.

###  4.1 VQA-MNIST

To analyze the impact of the degree of modularity of the library in systematic
generalization, we report results using the following libraries, which are
depicted in Figure 1a:

  * •

all - all - all: This is a library with a single image encoder module, a
single classifier module, and intermediate module defined as the Find module.
This library is commonly used in NMN [16], while the following ones are
explored for the first time in this paper.

  * •

all - group - group: This library contains a single image encoder, and
intermediate modules and classifiers separated per group. We divide sub-tasks
related to object categories, colors, spatial relations and so forth in
different groups.

  * •

group - all - all: This is a library with multiple image encoders separated
per group, where sub-tasks related to object categories, colors, spatial
relations and so forth belong to different groups. It has a single
intermediate module and a single classifier shared across all sub-tasks,
respectively.

  * •

group - group - group: This library has one image encoder module per each
group of sub-tasks, as in the image encoder of group - all - all. The
intermediate modules and the classifier also tackle groups of sub-task, as in
the intermediate modules and classifier of all - group - group.

  * •

sub-task - sub-task - sub-task: This is the library with the maximum degree of
modularity. The modules address a single sub-task at each stage of the NMN.

Other libraries of modules are analyzed in Appendix B, such as other state-of-
the-art NMN implementations and different libraries that help to isolate the
effect of the degree of modularity at the different stages. The analysis of
the libraries in the Appendix serves to strengthen the evidence that support
the conclusions obtained with the libraries analysed in this section.

---
|
(a) | (b)
|
(c) | (d)
Figure 3: _Results in VQA-MNIST._ Systematic generalization accuracy (referred
as OOD test accuracy in the plot) that evaluates VQA on novel combinations of
visual attributes, for different amount of training combinations (%), ie.
training data diversity. Results compare different libraries of modules in the
four types of tasks of VQA-MNIST: (a) attribute extraction from single object,
(b) attribute extraction from multiple objects, (c) attribute comparison
between pairs of separated objects, (d) comparison between spatial positions.

To measure systematic generalization accuracy, it needs to be guaranteed that
the novel combinations of attributes used for testing are not used in any
stage of learning [6]. We perform a grid-search over the NMNs’ hyper-
parameters, and evaluate the accuracy using an in-distribution validation
split at the last iteration of the training. The model with highest in-
distribution validation accuracy is selected, such that the novel combinations
of attributes are not used for hyper-parameter tuning (see Appendix B.2 for
more details).

Figure 3 shows the systematic generalization accuracy to novel combinations of
attributes on the four tasks in VQA-MNIST. Across all plots, each value on the
horizontal-axis displays the average percentage of training combinations for a
fixed $r$ (recall that $r$ is the minimum amount of times an attribute
instance appears in the training combinations) across two data-runs, where the
bars are grouped based on the value of $r$. In the legend, we specify the
library of modules used in the NMN. On the vertical-axis, we report mean and
standard deviation of the systematic generalization accuracy across the two
data-runs.

#### Systematic generalization compared to in-distribution generalization is
much more challenging.

Even though NMNs achieve the highest systematic generalization in the
literature [11, 5], Figure 3 shows that the systematic generalization accuracy
across VQA tasks can be very low when the amount of training combinations is
small. This highlights the difficulty to generalize to novel combinations of
attributes when the training datasets are biased. Recall that VQA-MNIST is a
synthetic dataset which could be considered one of the simplest. Yet, state-
of-the-art approaches in systematic generalization have a remarkable low
accuracy in this dataset. Appendix B.3 shows the in-distribution accuracy,
which is in stark contrast with the systematic generalization accuracy, as it
is close to $100\%$ accuracy in all evaluated cases. We underline that the low
systematic generalization accuracy depends on the amount of training
combinations, ie. the training data diversity, which has been reported in
other contexts beyond VQA, such as in object recognition [15, 14] and in
navigation tasks [8, 18].

#### Libraries with an intermediate degree of modularity, especially in the
image encoder stage, substantially improve systematic generalization.

The _group - all - all_ and the _group - group - group_ libraries achieve the
highest systematic generalization accuracy across all VQA tasks. These
libraries also clearly outperform the _sub-task - sub-task - sub-task_
library, which has the highest degree of modularity. The library with worst
systematic generalization is _all - all - all_ , which has the lowest amount
of modularity. Observe that the improvements of systematic generalization
accuracy are large in some cases (there are gaps of about $15\%$ accuracy
between libraries of modules). Also, note that the _group - all - all_ library
is clearly superior to _all - group - group_ , which suggests that a higher
degree of modularity is more helpful at the image encoder stage. Taking these
results together, we can conclude that a library with an intermediate degree
of modularity (_group_), specially at the image encoder stage, is crucially
important to improve systematic generalization for novel combinations of
visual attributes. In Appendix B, we report several control experiments that
support this conclusion. Namely, in Appendix B.4 we validate the conclusions
with other libraries of modules commonly used in the literature (such as _all
- sub-task - all_ , and variants with batch normalization applied to the _all_
libraries, which is when it is only possible). In Appendix B.5, B.6 and B.7 we
further validate the conclusion analysing variants for the image encoder,
intermediate modules and image classifier, respectively.

#### Higher systematic generalization is not reachable with larger amounts of
training examples.

Since using libraries with a larger number of modules may lead to modules
being trained with lower number of training examples, we controlled that our
conclusions are not affected by increasing the amount of training examples. We
consider the task of attribute extraction from single object and attribute
comparison between objects in separated images, and increased the training set
size of a factor ten (for a total of $2.1$M training examples), while keeping
the same percentage of training combinations of the previous experiments in
Figure 3 (a,c). Results are reported in Appendix B.8, which show that the
systematic generalization accuracy is not improved when increasing the number
of training examples, and conclusions are consistent to those in Figure 3
(a,c). Thus, the amount of training examples in these experiments already
achieves the highest possible systematic generalization accuracy by increasing
the dataset size, ie. our conclusions are not a consequence of having
libraries with data-starved modules.

###  4.2 SQOOP

We use the two libraries of modules used in previous work [11]: _all - all -
all_ and _all - sub-task - all_. To measure systematic generalization, we
consider the model at its last training iteration using the hyper-parameters
in previous works (batch normalization is used in the image encoder and
classifier stages).

Table 1: _Results in the SQOOP dataset._ Systematic generalization accuracy (%) for NMNs with tree program layout trained on SQOOP. Top row: SQOOP dataset with five objects per image, bottom row: SQOOP dataset with two objects per image. The _all - sub-task - all_ library reaches higher systematic generalization in the more difficult case of two objects per image. |  _all - all - all_ |  _all - sub-task - all_
---|---|---
Five objects per image
(as in previous work [11]) |  $\mathbf{99.8\pm 0.2}$ |  $\mathbf{99.96\pm 0.06}$
Two objects per image |  ${84\pm 2}$ |  $\mathbf{88.5\pm 0.5}$

In Table 1, we show the systematic generalization accuracy (mean and standard
deviation across five trials). Results are consistent with the conclusions
obtained in VQA-MNIST. On the original SQOOP dataset with five objects per
image, both libraries achieve almost perfect systematic generalization over
novel test questions. For the dataset with two objects in the image, the
dataset is more biased because of the lower amount of co-occurring objects,
and hence, it increases the difficulty of generalizing to novel question. In
this case, the _all - sub-task - all_ library achieves significantly higher
systematic generalization performance than the _all - all - all_ library.
Finally, in Appendix C we report the performance of other VQA models that do
not use modules (ie. FiLM [19] and MAC [20, 11]). The results provide re-
assurance that NMNs achieve higher systematic generalization.

##  5 Application on CLEVR-CoGenT Split for Systematic Generalization

In this section, we apply the findings in the previous section on existing NMN
architectures used in practice. To do so, we use CLEVR, a diagnostic dataset
in VQA [2]. This dataset consists of complex 3D scenes with multiple objects
and ground-truth program layouts formed by compositions of sub-tasks. This
dataset comes with additional splits to test systematic generalization, namely
the Compositional Generalization Test (CLEVR-CoGenT). CLEVR-CoGenT is divided
in two conditions where cubes and cylinders appear in a limited amount of
colors, that are inverted between training and testing (see Appendix D.1 for
details). In this way, we can measure systematic generalization to novel
combinations of visual attributes (shape and color in this case).

We now show that the analysis in previous sections is helpful to improve
state-of-the-art NMNs. We focus on Vector-NMN, which recently has been shown
to outperform all previous NMNs [5]. Vector-NMN uses a single image encoder
shared across all sub-tasks. The Vector-NMN’s image encoder takes as input the
features extracted from a pre-trained ResNet-101 [21], and consists of two
trainable convolutional layers shared among all sub-tasks. The details of
Vector-NMN definition can be found in Appendix D.2. Our variant of Vector-NMN
is based on tuning the degree of modularity in the encoder stage, as we have
previously shown this can substantially improve the systematic generalization
capabilities. Thus, we modify the library of Vector-NMN by using a higher
degree of modularity at the image encoder stage, ie. one module for each group
of sub-tasks. Concretely, we define the groups as: (i) counting tasks, (ii)
tasks related to colors, (iii) materials, (iv) shapes, (v) sizes, (vi) spatial
relations, (vii) logical operations, (viii) input scene (for details, see
Appendix D.3).

Table 2: _Results in the CLEVR-CoGenT dataset._ Mean and standard deviation of test accuracy (%) across five repetitions. NMNs are tested on in-distribution and out-of-distribution (systematic generalization). |  Tensor-NMN
with _all_
image encoder |  Vector-NMN
with _all_
image encoder |  Vector-NMN
with _group_
image encoder (ours)
---|---|---|---
in-distribution |  $97.9\pm 0.1$ |  $\mathbf{98.0\pm 0.2}$ |  $94.4\pm 0.3$
syst. generalization |  $72.7\pm 0.5$ |  $73.2\pm 0.2$ |  $\mathbf{77.3\pm 1.3}$

The hyper-parameters are fixed to the ones in the previous work [5], and the
ground-truth program layout is given. Tables 2 and 3 show the mean and
standard deviation accuracy (across five runs with different network
initialization each) of the original Vector-NMN, our version of Vector-NMN
with a more modular (_group_) image encoder, and also Tensor-NMN, which is
another NMN commonly used in the literature that here serves as baseline.
Table 2 shows the in-distribution accuracy and the systematic generalization
accuracy in objects with novel combinations of shape and color. Results shows
the wide gap between in-distribution and out-of-distribution for all NMNs. Our
Vector-NMN with a more modular (_group_) image encoder achieves the best
systematic generalization performance. This comes with a reduction of in-
distribution generalization, which can be explained by the trade-off between
in-distribution and out-of-distribution generalization reported in previous
works [14, 22].

Table 3 shows the break-down of systematic generalization for each type of
question. Note that the systematic generalization accuracy varies depending on
questions types. The questions that are mostly affected by the limited amount
of object shapes and color combinations in CLEVR-CoGenT, ie. query_shape and
query_color, is where a higher degree of modularity in the image encoder
brings an improvement of the accuracy of $12\%$ and $7\%$, respectively. A
broader comparison of our approach with other non-modular approaches such as
FiLM [19] and MAC [20, 5], and other NMN variants is reported in Appendix D.4.
These results further demonstrate the higher systematic generalization
accuracy of our approach.

Table 3: _Breakdown of the results in the CLEVR-CoGenT dataset._ Breakdown by question type. Systematic generalization accuracy (%) is reported, and it is the average across five trials. |  Tensor-NMN
with _all_
image encoder |  Vector-NMN
with _all_
image encoder |  Vector-NMN
with _group_
image encoder (ours)
---|---|---|---
count |  $69.7\pm 0.8$ |  $70.4\pm 0.4$ |  $\mathbf{71\pm 1}$
equal_color |  $75.6\pm 0.8$ |  $74\pm 1$ |  $\mathbf{80\pm 1}$
equal_integer |  $82.7\pm 0.3$ |  $78\pm 2$ |  $\mathbf{85\pm 2}$
equal_material |  $74\pm 2$ |  $74.2\pm 0.7$ |  $\mathbf{84\pm 2}$
equal_shape |  $\mathbf{91\pm 2}$ |  $89\pm 3$ |  $79\pm 2$
equal_size |  $75\pm 1$ |  $75\pm 1$ |  $\mathbf{88\pm 2}$
exist |  $84.2\pm 0.4$ |  $\mathbf{84.4\pm 0.4}$ |  $84.4\pm 0.5$
greater_than |  $83.8\pm 0.6$ |  $83.6\pm 0.4$ |  $\mathbf{89\pm 1}$
less_than |  $80.7\pm 0.9$ |  $82.0\pm 0.5$ |  $\mathbf{87\pm 2}$
query_color |  $58\pm 1$ |  $60\pm 1$ |  $\mathbf{67\pm 4}$
query_material |  $84.1\pm 0.9$ |  $84.7\pm 0.4$ |  $\mathbf{88.2\pm 0.8}$
query_shape |  $37\pm 1$ |  $40\pm 3$ |  $\mathbf{52\pm 3}$
query_size |  $83.5\pm 0.6$ |  $84.7\pm 0.7$ |  $\mathbf{89.5\pm 0.5}$

##  6 Conclusions and Discussion

Our results demonstrate that NMNs with a library of modules with an
intermediate degree of modularity, especially at the image encoder stage,
substantially improves systematic generalization. This finding is consistent
across datasets of different complexity and NMN architectures, and it is
easily applicable to improve state-of-the-art NMNs by using modular image
encoders.

This work also has led to new research questions. While we have shown that
modularity has a large impact in systematic generalization, we have tuned the
degree of modularity at hand and in practical applications it is desirable to
do this automatically. Also, it is unclear how other types of bias, such as
bias in the program layout, could affect systematic generalization. To
motivate follow-up work in other types of bias beyond the ones analyzed in
this work, in Appendix E we report a pilot experiment that shows that bias in
the program layout could degrade systematic generalization of the NMNs we
introduced. This suggests that there may be a trade-off between the degree of
modularity and bias in the program layout, which will be investigated in
future works.

Finally, we are intrigued about the neural mechanisms that facilitate
systematic generalization and how the degree of modularity affects those
mechanisms. Some hints towards an answer have been pointed out by [14], which
shows that the emergence of invariance to nuisance factors at the individual
neuron level improves systematic generalization for object recognition.

## Acknowledgments and Disclosure of Funding

We would like to thank Moyuru Yamada, Hisanao Akima, Pawan Sinha and Tomaso
Poggio for useful discussions and insightful advice provided during this
project, and Ramdas Pillai and Atsushi Kajita from Fixstars Solutions for
their assistance executing the large scale computational experiments. This
work has been supported by the Center for Brains, Minds and Machines (funded
by NSF STC award CCF-1231216), the R01EY020517 grant from the National Eye
Institute (NIH) and Fujitsu Limited (Contract No. 40008819 and 40009105).

## Code and Data Availability

Code and data can be found in the following github repository:
https://github.com/vanessadamario/understanding_reasoning.git.

## References

  * [1] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual Genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123(1):32–73, 2017.
  * [2] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2017), pages 2901–2910, 2017.
  * [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  VQA: Visual question answering.  In Proceedings of the 15th IEEE International Conference on Computer Vision (ICCV 2015), pages 2425–2433, 2015.
  * [4] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), pages 4971–4980, 2018.
  * [5] Dzmitry Bahdanau, Harm de Vries, Timothy J O’Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville.  CLOSURE: Assessing systematic generalization of CLEVR models.  arXiv preprint arXiv:1912.05783v2, 2020.
  * [6] Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel.  On the value of out-of-distribution testing: An example of Goodhart’s law.  In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 407–417, 2020.
  * [7] Kushal Kafle and Christopher Kanan.  Visual question answering: Datasets, algorithms, and future challenges.  Computer Vision and Image Understanding, 163:3–20, 2017.
  * [8] Brenden Lake and Marco Baroni.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In Proceedings of the the 35th International Conference on Machine Learning (ICML 2018), pages 2873–2882, 2018.
  * [9] João Loula, Marco Baroni, and Brenden Lake.  Rearranging the familiar: Testing compositional generalization in recurrent networks.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 108–114, 2018.
  * [10] Jerry A Fodor and Zenon W Pylyshyn.  Connectionism and cognitive architecture: A critical analysis.  Cognition, 28(1-2):3–71, 1988.
  * [11] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville.  Systematic generalization: What is required and can it be learned?  In Proceedings of the 7th International Conference on Learning Representations (ICLR 2019), 2019.
  * [12] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein.  Neural module networks.  In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pages 39–48, 2016.
  * [13] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Inferring and executing programs for visual reasoning.  In Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV 2017), pages 2989–2998, 2017.
  * [14] Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix.  When and how do CNNs generalize to out-of-distribution category-viewpoint combinations?  arXiv preprint arXiv:2007.08032v2, 2021.
  * [15] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc’Aurelio Ranzato.  Task-driven modular networks for zero-shot compositional learning.  In Proceedings of the 17th IEEE/CVF International Conference on Computer Vision (ICCV 2019), pages 3593–3602, 2019.
  * [16] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko.  Learning to reason: End-to-end module networks for visual question answering.  In Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV 2017), pages 804–813, 2017.
  * [17] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [18] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake.  A benchmark for systematic generalization in grounded language understanding.  In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 19861–19872, 2020.
  * [19] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  FiLM: Visual reasoning with a general conditioning layer.  In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18), 2018.
  * [20] Drew A Hudson and Christopher D Manning.  Compositional attention networks for machine reasoning.  In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), 2018.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pages 770–778, 2016.
  * [22] Syed Suleman Abbas Zaidi, Xavier Boix, Neeraj Prasad, Sharon Gilad-Gutnick, Shlomit Ben-Ami, and Pawan Sinha.  Is robustness to transformations driven by invariant neural representations?  arXiv preprint arXiv:2007.00112, 2020.

##  Appendix A VQA-MNIST Dataset

In Appendix A.1, we describe the attributes and the comparisons in the VQA-
MNIST datasets. The algorithm for the generation of the training and testing
combinations is in Appendix A.2. Given the combinations, in Appendix A.3 we
define the procedure to generate training and testing examples.

###  A.1 Attributes and Comparisons

#### Visual attributes.

Each object in VQA-MNIST is characterized by a category, the original MNIST
category from 0 to 9, a color among red, green, blue, yellow and pink, a light
condition among bright, half-bright, and dark, correspondent to a
multiplicative factor $(1,\ 0.7,\ 0.4)$ applied on the whole image, and a size
among large, medium, small, correspondent to three scale factors $(1,\ 5/7,\
1/2)$ of the original digit. This corresponds to a total of 21 attribute
instances. Figure App.1 shows some objects with a reduced amount of
combination of attributes.

Figure App.1: VQA-MNIST: some examples of training images, with a limited
amount of attributes combinations.

#### Attribute comparisons.

The sub-tasks used for attribute comparison between two separated objects are
in Table App.1. The sub-tasks for comparison of spatial relations are in Table
App.2.

Table App.1: Sub-tasks for attribute comparison between pairs of separated objects. The sub-tasks are divided in groups. _group of sub-tasks_ | _sub-tasks_
---|---
category | category
color | color
light | light
size | size
comparison of category |  same_category; different_category
comparison of color |  same_color; different_color
comparison of light condition |  same_light; different_light; brighter; darker
comparison of size |  same_size; different_size; larger; smaller
Table App.2: Sub-tasks for comparison of spatial positions between two objects. The sub-tasks are divided in groups. _group of sub-tasks_ | _sub-tasks_
---|---
category |  0; 1; 2; 3; 4; 5; 6; 7; 8; 9
color |  red; green; blue; yellow; pink
light |  bright; half-bright; dark
size |  large; medium; small
spatial relations |  left_of; right_of; above; below

###  A.2 Algorithm for the Generation of Training and Test Combinations

To generate the training and test combinations across all the VQA-MNIST
datasets, we fix the set of attribute types: category (attr1), color (attr2),
light condition (attr3), and size (attr4); and we define a list of attribute
instances for each type. The list $\mathrm{attr1}=[\texttt{0,\ldots, 9}]$
contains the ten categories, the list $\mathrm{attr2}=[\mathrm{\texttt{red,
green, blue, yellow, pink}}]$ the five colors, the list
$\mathrm{attr3}=[\mathrm{\texttt{bright, half-bright, dark}}]$ the three light
conditions, and the list $\mathrm{attr4}=[\mathrm{\texttt{small, medium,
large}}]$ the three sizes. We define a single comprehensive list of all
attributes as
$\mathrm{attributes}=\mathrm{attr1}+\mathrm{attr2}+\mathrm{attr3}+\mathrm{attr4}$,
of length 21. Two integers are required for the generation of training and
test combinations: $\mathrm{n\\_combinations\\_train}$ and
$\mathrm{n\\_combinations\\_test}$, which define the minimum amount of times
(indicated with the $r$ parameter) each attribute will appear in the training
and testing sets, respectively. We also define $\mathrm{train\\_combinations}$
and $\mathrm{test\\_combinations}$ as two empty lists. Through a stochastic
procedure described in Algorithm 1, we fill these lists with the objects for
the training and the testing sets.

1:$\mathrm{n\\_train\\_combinations},\ \mathrm{n\\_test\\_combinations}$

2:$\mathrm{train\\_combinations,\ test\\_combinations}=[\ ],[\ ]$, $k=0$,
$\mathrm{random\\_seed}=n$

3:for $k<\mathrm{n\\_test\\_combinations}$ do

4:

5: $\mathrm{taken}=\mathrm{dict}()$

6: for $a\\_$ in $\mathrm{attributes}$ do

7: $\mathrm{taken}[a\\_]=False$

8: end for$\triangleright$ At every iteration in the loop we set the
dictionary of seen attribute to False

9:

10: $\mathrm{missing\\_attributes}=True$

11: while $\mathrm{missing\\_attributes}$ do $\triangleright$ Until we see all
the attributes at the $k$-th iteration

12:
$\mathrm{candidate}=[\mathrm{choice}(\mathrm{attr1}),\mathrm{choice}(\mathrm{attr2}),\mathrm{choice}(\mathrm{attr3}),\mathrm{choice}(\mathrm{attr4})]$
$\triangleright$ random extraction using uniform distribution

13: $\mathrm{exists=candidate\ in\ test\\_combinations}$

14: $\mathrm{reject}=True$ $\triangleright$ We assume that all attributes in
candidate already appeared

15: for $c\\_$ in $\mathrm{candidate}$ do

16: $\mathrm{reject}\ *=\mathrm{taken}[c\\_]$

17: end for

18: if $\mathrm{not\ reject\ \&not\ exists}$ then

19: $\mathrm{test\\_combinations}\leftarrow\mathrm{candidate}$

20: for $c\\_$ in $\mathrm{candidate}$ do

21: $\mathrm{taken}[c\\_]=True$

22: end for

23: end if

24: if all values of $\mathrm{taken}$ are _True_ then

25: $\mathrm{missing\\_attributes}\leftarrow False$

26: $k\leftarrow k+1$ $\triangleright$ All attributes appeared once, next
iteration

27: end if

28: end while

29:end for

30:To fill the list $\mathrm{train\\_combinations}$, substitute in line 2
$\mathrm{n\\_test\\_combinations}$ to $\mathrm{n\\_train\\_combinations}$.
Change declaration of line 12 into line 30. Repeat from 2-28.

31:$\mathrm{exists}$ = ($\mathrm{candidate}$ in
$\mathrm{test\\_combinations}$) OR ($\mathrm{candidate}$ in
$\mathrm{train\\_combinations}$)

Algorithm 1 Generation of Training and Test Combinations

The additional variable $\mathrm{reject}$ and the loop in 14-16 is to exclude
a new tuple if it contains a list of attributes which have already appeared,
_e.g.,_ if $\mathrm{test\\_combinations}=[(3,\mathrm{yellow,bright,small}),\
(4,\mathrm{blue,dark,large})]$, the candidate $(3,\mathrm{blue,dark,small})$
will be rejected. This constraint allows to guarantee that all attributes
appear at least $r$ times while minimizing the number of combinations used to
do so.

###  A.3 Procedure for the Generation of Training and Test Examples

Across all the VQA-MNIST datasets, we leverage on Algorithm 1 to generate the
combinations of attributes. The digits of MNIST used for the training,
validation, and test splits have been fixed a priori, with $50$K training
digits, $10$K validation digits, and the $10$K test digits of the original
MNIST. Depending on the VQA task, we differentiate the data generation
process.

#### Attribute Extraction.

Given a split (training or test), and the VQA question with its corresponding
sub-task, we divide the objects in the split into those providing a positive
and negative example. We randomly select a digit from the MNIST split which
belongs to the category of the example, and we introduce the additional
attributes to the image. We generate evenly examples for both cases, so to
have a balanced dataset. The program layout corresponds to one of the 21
attribute instances.

#### Attribute Comparison.

The data generation repeats identically at training and testing. Given a VQA
question about a relation and a split, we first collect all the pairs of
objects from that split that provide a positive answer to the question. We
repeat the same for the negative pairs. Then, we generate the examples based
on the attributes for the pair of objects, by extracting images from the MNIST
split which match with the categories of the objects. We generate an even
number of positive and negative examples for each question, to have a balanced
dataset.

For the tasks of attribute comparison, the program layout has a tree
structure. For attribute comparison between pairs of separated objects, the
sub-tasks at the leaves can be category, color, light condition, and size.
These sub-tasks match with the relational questions (eg., if we are asking if
two objects are the same color, the leaves tackle the sub-task color). The
sub-tasks and their division in groups are detailed in Table App.1. For
comparison between spatial positions, the sub-tasks at the leaves can assume
the form of one of the 21 attribute instances, while the root coincides with
one of the four 2D spatial relations among below, above, left, and right. The
sub-tasks and their division in groups are detailed in Table App.2.

##  Appendix B Implementation Details and Supplemental Results on VQA-MNIST

In Section B.1, we give the definition of image encoder and classifier
architectures, common to all the NMNs trained on VQA-MNIST. We present the
experimental setup and optimization details for the experiments on VQA-MNIST
in Section B.2. The in-distribution generalization of the main libraries on
VQA-MNIST are in Section B.3.

Then, we introduce a series of control libraries to prove the generality of
our findings. In particular, in Section B.4 we compare libraries with modular
image encoder to state-of-the-art libraries in the NMN literature. In Section
B.5 we analyze the effect of different degrees of modularity at the image
encoder stage. In Section B.6, we further verify the impact of different
architectures for the intermediate modules on systematic generalization. In
Section B.7, we analyze the effect of different degrees of modularity at the
classifier stage. Finally, in Section B.8, we report the systematic
generalization of NMNs trained ten times larger datasets than those in Figure
3.

###  B.1 Description of the Image Encoder and Classifier Architectures

Table App.3: Image encoder module VQA-MNIST. The first two parameters in
Conv2d refer to the number of input and output channels. For those libraries
without batch normalization, the BatchNorm2d and BatchNorm1d should not be
considered. Image Encoder
---
| Layer | Parameters
(0) | Conv2d | (3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1) | BatchNorm2d | (64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2) | ReLU |
(3) | Conv2d | (64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(4) | BatchNorm2d | (64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5) | ReLU |
(6) | MaxPool2d | (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(7) | Conv2d | (64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8) | BatchNorm2d | (64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(9) | ReLU |
(10) | Conv2d | (64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11) | BatchNorm2d | (64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(12) | ReLU |
(13) | MaxPool2d | (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(14) | Conv2d | (64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(15) | BatchNorm2d | (64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(16) | ReLU |
(17) | Conv2d | (64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(18) | BatchNorm2d | (64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(19) | ReLU |
Table App.4: Classifier module for libraries VQA-MNIST. The first two
parameters in Conv2d refer to the number of input and output channels. For
those libraries without batch normalization, the BatchNorm2d and BatchNorm1d
should not be considered. Binary Classifier
---
| Layer | Parameters
(0) | Conv2d | (64, 512, kernel_size=(1, 1), stride=(1, 1))
(1) | BatchNorm2d | (512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2) | ReLU |
(3) | MaxPool2d | (kernel_size=7, stride=7, padding=0, dilation=1, ceil_mode=False)
(4) | Flatten |
(5) | Linear | (in_features=512, out_features=1024, bias=True)
(6) | BatchNorm1d | (1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(7) | ReLU |
(8) | Linear | (in_features=1024, out_features=2, bias=True)

The architectures of the image encoder and the classifier modules are shared
across all the experiments on VQA-MNIST. All the libraries in Section 4, as
well as most of those in this section, do not have batch normalization, except
for _all(bn) - all - all(bn)_ and _all(bn) - sub-task - all(bn)_ libraries. We
report the implementation of an image encoder module and a classifier module,
respectively in Table App.3 and Table App.4, where we include the batch
normalization.

For those image encoders and classifiers without batch normalization, the
BatchNorm2d and BatchNorm1d layers need to be excluded (respectively
transformations 1, 4, 8, 11, 15, and 18 in the image encoder and
transformations 1 and 6 in the classifier). Notice that in Tables App.3 and
App.4, the first two parameters for the Conv2d define the number of input and
output channels.

###  B.2 Optimization Details

Across all the experiments on VQA-MNIST on $210K$ training examples, we fix
the number of training steps to $200K$. For the task of attribute extraction
from a single object, the grid of hyper-parameters includes batch size with
candidate values $[128,\ 256]$, and the learning rates with values $[10^{-4},\
10^{-3},\ 0.005,\ 0.01]$. Given the high computational cost of training, for
all the remaining experiments we reduce the number of models by fixing the
batch size to $64$ and the learning rate hyper-parameters to the values
$[10^{-5},10^{-4},10^{-3},0.005,0.01]$.

During training, the learning rate is kept constant. We notice that in some
cases, especially on comparison tasks, the training loss function, as well as
the training accuracy, often presents several plateaus alternated to jumps,
despite the absence of a decaying learning rate. This did not allow us to use
early stopping as a strategy to accelerate the experiments.

###  B.3 In-distribution generalization

---
|
(a) | (b)
|
(c) | (d)
Figure App.2: In-distribution generalization on the VQA-MNIST dataset, for (a)
attribute extraction from single object, (b) attribute extraction from
multiple objects, (c) attribute comparison between pairs of separated objects,
and (d) comparison between spatial positions.

We aim at comparing the in-distribution and systematic generalization trends
for the libraries previously tested on novel combinations, in Figure 3. The
plots in Figure App.2 shows the in-distribution generalization for (a)
attribute extraction from single object, (b) attribute extraction from
multiple objects, (c) attribute comparison between pairs of separated objects,
and (d) comparison between spatial positions. The disparity between testing
in-distribution generalization or systematic generalization is dramatic. In
absence of a bias, the tasks of attribute extraction and comparison, with the
given amount of training examples looks easy. Indeed, the plots across the
four VQA tasks show that all the libraries represent equivalently a good
choice to achieve generalization within in-distribution testing examples. This
result emphasizes the importance of measuring systematic generalization as a
strategy to highlight the limited capability of models to generalize to novel
combinations.

###  B.4 Results for Additional state-of-the-art Libraries and Comparison with
Libraries with Modular Image Encoder

---
(a)
|
(b) | (c)
|
(d) | (e)
Figure App.3: Comparison of libraries with modular image encoder and state-of-
the-art libraries. OOD test accuracy refers to systematic generalization
accuracy. (a) State-of-the-art libraries and library with modular image
encoder (_group - all - all_). Systematic generalization accuracy on (b)
attribute extraction from single object, (c) attribute extraction from
multiple objects, (d) attribute comparison between pairs of separated objects,
and (e) comparison between spatial positions.

With the exception of the _all - all - all_ library, the libraries in Figure
1a are not the standard ones from the literature. To confirm our findings on
the superiority of the modular image encoder (_group - all - all_ in
particular) with previous works, we add the _all - sub-task - all_ library
used in [13]. Its implementation consists of a single image encoder and a
single classifier, and Residual intermediate modules. Given the usage of a
single module at the image encoder and the classifier stages, _all - all -
all_ library and _all - sub-task - all_ library have variants with batch
normalization at both the image encoder and classifier [11]. We denote these
libraries as _all(bn) - all - all(bn)_ and _all(bn) - sub-task - all(bn)_. See
Figure App.3a for the depictions of these libraries of modules.

We measure the systematic generalization of those libraries. In Figure
App.3b-e, we reported the performance on (b) attribute extraction from single
object, (c) attribute extraction from multiple objects, (d) attribute
comparison between pairs of separated objects, and (e) comparison between
spatial positions. The systematic generalization accuracy for state-of-the-art
methods remains clearly below the performance of the _group - all - all_
library.

###  B.5 Results for Libraries with Modular Image Encoder

Results in Figure 3 highlight the importance of a library of modules at the
image encoder stage, together with a medium degree of modularity. To further
investigate this aspect, we introduce the _sub-task - all - all_ library, an
additional library with image encoder modules per each sub-task. We further
define two additional libraries for the task of attribute comparison, reported
in Figure App.4b. There, given the tree program layout for this task, we rely
on libraries with different degrees of modularity at the intermediate modules
stage: at the leaves, we use a module per sub-task, while the root has an
intermediate module per group of sub-task, where the division of sub-tasks
follows the one in Table App.1. The main difference in the libraries of Figure
App.4b is at the image encoder stage. On the left, the _all - group/sub-task -
all_ library relies on a single image encoder module. On the right, the _sub-
task - group/sub-task - all_ library relies on the largest library of image
encoder modules. These two libraries have a single classifier module.

---
(a)
(b)
Figure App.4: Comparison between libraries with different degrees of
modularity at the image encoder stage. (a) Libraries with different degrees of
modularity at the image encoder stage, (b) Implementation of libraries with
group/sub-task libraries at intermediate module stage.
---
|
(a) | (b)
|
(c) | (d)
Figure App.5: Results for comparison between libraries with different degree
of modularity at the image encoder stage. OOD test accuracy refers to
systematic generalization accuracy. (a-d) Systematic generalization accuracy
on (a) attribute extraction from single object, (b) attribute extraction from
multiple objects, (c) attribute comparison between pairs of separated objects,
and (d) comparison between spatial positions.

We tested the libraries in the first three columns of Figure App.4a on the
VQA-MNIST tasks, while the libraries on the last two columns are used on the
task of attribute comparison between a pair of separated objects. In Figure
App.5a-d, we reported the systematic generalization performance for the tasks
of (a) attribute extraction from single object, (b) attribute extraction from
multiple objects, (c) attribute comparison between pairs of separated objects,
and (d) comparison between spatial positions. We observe that the _group - all
- all_ library consistently outperforms the _sub-task - all - all_ library
across all the tasks. The _sub-task - all - all_ library is nonetheless
superior to the standard _all - all - all_ library. In (c), the modularity of
the image encoder library brings advantage also to the network with different
degrees of modularity at the intermediate stage.

###  B.6 Results for Libraries for Different Implementations of Intermediate
Modules per sub-task

---
(a)
|
(b) | (c)
|
(d) | (e)
Figure App.6: Comparison between libraries of same degree of modularity, but
different implementation of the intermediate modules. OOD test accuracy refers
to systematic generalization accuracy. (a) Two libraries with different
implementation of the intermediate modules and _all - all - all_ and _group -
all - all_ libraries for reference. Systematic generalization accuracy on (b)
attribute extraction from single object, (c) attribute extraction from
multiple objects, (d) attribute comparison between pairs of separated objects,
and (e) comparison between spatial positions.

To exclude that the different implementation of the Residual from the Find
module could be a reason for the poorer systematic generalization performance
of the _sub-task - sub-task - sub-task_ library, we introduce a similar
library, which we call _sub-task - sub-task(Find) - sub-task_ library. In the
latter, the implementation of the intermediate modules rely on the Find module
as in Equations 5-6, where each sub-task defines a group. See Figure App.6a
for the depictions of these libraries of modules.

Figure App.6b-e shows the systematic generalization of the libraries of Figure
App.6a, trained on (b) attribute extraction from single object, (c) attribute
extraction from multiple objects, (d) attribute comparison between pairs of
separated objects, and (e) comparison between spatial positions. We observe
that libraries with sub-task degree of modularity at the image encoder stage
perform quite similar independently from the implementation of their
intermediate modules, and show consistently lower performance than the _group
- all - all_ library.

###  B.7 Results for Libraries with Modular Classifier

---
(a)
|
(b) | (c)
|
(d) | (e)
Figure App.7: Comparison between libraries with different degrees of
modularity at the classifier stage. OOD test accuracy refers to systematic
generalization accuracy. (a) Three libraries with different degrees of
modularity at the classifier stage and the _group - all - all_ library for
reference. Systematic generalization accuracy on (b) attribute extraction from
single object, (c) attribute extraction from multiple objects, (d) attribute
comparison between pairs of separated objects, and (e) comparison between
spatial positions.

We further consider the effect of libraries with different degrees of
modularity at the classifier stage on systematic generalization. To do so, we
introduce two libraries with a shared image encoder module and shared
intermediate modules. As before, we envision three degrees of modularity at
the classifier stage: shared module, that is the _all - all - all_ library,
per group of sub-tasks, that is, the _all - all - group_ library, and per sub-
task, that is, the _all - all - sub-task_ library. Figure App.7a depicts these
libraries.

The systematic generalization performance for the libraries in Figure App.7a
across the VQA-MNIST tasks is reported in Figure App.7b-e. The conclusion here
is that the degree of modularity of the library at the classifier stage alone
does not bring any advantages across the tested tasks.

###  B.8 Results on More Training Examples

We performed additional experiments to check if the amount of training
examples plays a role in the trade-off among degrees of modularity. With this
goal, we increased by a factor ten the amount of training examples, in Figure
App.8, for (a) attribute extraction from single object, and (b) attribute
comparison between pairs of separated objects.

The NMNs trained on (a) have a fixed batch size to value 64 and a grid of
learning rates with values $[10^{-5},10^{-4},0.001,0.005,0.01]$. We fixed at
half a million the amount of iterations. The NMNs trained on dataset (b) share
the same set of hyper-parameters of the case with ten times less data. We
fixed the number of iterations to one million, but due to the high
computational cost, we ran the experiments for maximum $\sim 300$ hours on a
NVIDIA DGX-1 machine. Not all the models reached the last iteration, but all
their loss curves qualitatively show convergence. We report the results in
Figure App.8. (a) shows the case of attribute extraction from a single
objects, while (b) reports the performance for attribute comparison between
pairs of objects.

---
|
(a) | (b)
Figure App.8: Experiments with ten times more data. OOD test accuracy refers
to systematic generalization accuracy. The colored bars correspond to networks
trained with $2.1$ million images, dots correspond to the case of $210K$
images. Systematic generalization accuracy for (a) attribute extraction from
single object, (b) attribute comparison between pairs of separated objects.

The colored bars report the systematic generalization performance over one
trial, with $2.1$ million training examples, while the black dots display the
systematic generalization performance for the corresponding model trained on
ten times less examples. We observe that, consistently with the previous
results, the _group - group - group_ library outperforms the other libraries,
in Figure App.8a. In Figure App.8b, the NMN with modular library at the image
encoder stage achieve the highest systematic generalization. This result shows
that high systematic generalization is not reachable by increasing the amount
of training data. This conclusion is consistent with the observation in
previous works on out-of-distribution generalization [9, 14].

##  Appendix C Supplemental Results on SQOOP

To perform these experiments, we rely on the networks proposed in [11]. Both
tree-NMNs make use of batch normalization in their image encoder and
classifier stages.

To validate that NMNs outperform other VQA models in systematic
generalization, we report in Table App.5 the results obtained on SQOOP for two
non-modular neural networks, namely re-implementation of MAC [20] from
Bahdanau _et al._ [11] and FiLM [19], with the results of tree-NMNs we have
reported in Table 1. We adopted the same hyper-parameters from [11].

For each model, we observe a wide gap in systematic generalization on the two
datasets. For the FiLM architecture, reducing the amount of objects in the
image leads to higher systematic generalization than for the case of five
objects, while the opposite happens to MAC.

A direct comparison between non-modular networks and NMN is not possible,
given that the NMN are provided with an optimal program layout. Nonetheless,
the higher systematic generalization of NMNs highlights the potential benefit
in the use of NMNs wherever there is a mechanism to identify a proper program,
given a question.

Table App.5: Systematic generalization accuracy (%) for MAC (reimplemented by Bahdanau _et al._ [11]), FiLM, and NMN architectures with tree layouts. Top row: bias in the question (five objects per image). Bottom row: bias in the co-occurrence of objects in the image (two objects per image). Bias type |  MAC
(Bahdanau _et al._ [11]) |  FiLM |  _all - all - all_
with bn |  _all - sub-task - all_
with bn
---|---|---|---|---
Five objects per image
(as in [11]) |  $84\pm 4$ |  $70\pm 2$ |  $\mathbf{99.8\pm 0.2}$ |  $\mathbf{99.96\pm 0.06}$
Two objects per image |  $66\pm 5$ |  $77\pm 3$ |  ${84\pm 2}$ |  $\mathbf{88.5\pm 0.5}$

##  Appendix D Implementation Details and Supplemental Experiments on CLEVR-
CoGenT

In Appendix D.2 we report the implementation of the Vector-NMN as in [5], with
its image encoder, intermediate module, and classifier. Then, in Appendix D.1
we describe the Compositional Generalization Test (CoGenT), its tasks, and our
grouping of sub-tasks, in Appendix D.3. Lastly, we report the results obtained
on single questions for other non-modular networks, and different NMN
implementations.

###  D.1 CLEVR-CoGenT

The Compositional Generalization Test (CLEVR-CoGenT) dataset [2] includes
training, validation, and test splits. The data generation process follows the
usual criteria of CLEVR, but CoGenT is built to explicitly measure the ability
of a model to generalize to novel combinations of objects and attributes.
CLEVR-CoGenT has two conditions: Condition A and Condition B. In Condition A,
the one used at training, cubes, cylinders and spheres appear in different
sizes, colors, and positions. Cubes are gray, blue, brown, or yellow,
cylinders are red, green, purple, or cyan, while spheres can have any color.
In Condition B these combinations of attributes are reversed. Cubes are red,
green, purple, or cyan, cylinders are gray, blue, brown, or yellow, and
spheres can have any color. Condition B is the split to measure systematic
generalization of a model. The models can be validated and tested in-
distribution as well.

###  D.2 Vector-NMN

We define Vector-NMN as in previous work [5]. On the family of CLEVR datasets,
the image encoder of Vector-NMN takes in input the output features from a
ResNet-101 as detailed in Section 4.1 of [2]. The image encoder consists of
two convolutional layers with ReLU activation, as detailed in Table App.6, the
classifier is a multi-layer perceptron, as detailed in Table App.7.

Table App.6: Image encoder of Vector-NMN. Image encoder
---
(0) | Conv2d | ((1024, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1) | ReLU |
(2) | Conv2d | (64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(3) | ReLU |
Table App.7: Classifier of Vector-NMN. Classifier
---
(0) | Flatten |
(1) | Linear | (in_features=128, out_features=1024, bias=True)
(2) | ReLU |
(3) | Linear | (in_features=1024, out_features=32, bias=True)

The Vector-NMN leverages on modulation through a set of weights, similarly to
the FiLM network [19]. Every intermediate module of the Vector-NMN receives in
input the concatenation of the embedding for the sub-task and the output from
previous modules, defined as $h_{c}=[\text{Emb}(k);s_{x};s_{y}]$. We define
the output of the image encoder given the input features of an image as
$s_{\text{img}}$. In the following, we report the definition of Vector-NMN
[5]:

$\displaystyle[\beta_{j},\tilde{\gamma}_{j}]=W_{2}^{j}(\text{ReLU}(W_{1}^{j}h_{c}+b_{1}^{j})+b_{2}^{j}),\
\text{ with }j\in[0,1],$ (App.1)
$\displaystyle\gamma_{j}=2\tanh{(\tilde{\gamma}_{j})}+1,$ (App.2)
$\displaystyle\tilde{h}_{1}=\text{ReLU}(U_{1}*(\gamma_{1}\odot
s_{\text{img}}\oplus\beta_{1})),$ (App.3)
$\displaystyle\tilde{h}_{2}=\text{ReLU}(U_{2}*(\gamma_{2}\odot\tilde{h}_{1}\oplus\beta_{2})+h_{x}),$
(App.4) $\displaystyle h_{i}(k,s_{x},s_{y})=\text{maxpool}(\tilde{h}_{2}).$
(App.5)

###  D.3 Grouping of sub-tasks

We decide to divide sub-tasks based on their similarity. We group together all
the sub-tasks which require determining colors, shapes, materials, etc. We
identify eight groups of sub-tasks. In the following, we specify each group,
each with its related sub-tasks: counting, with count, equal_integer, exist,
greater_than, less_than, unique; colors, with equal_color, filter_color[*color
instance], query_color, same_color; materials, with equal_material,
same_material, filter_material[*material instance], query_material; shapes,
with equal_shape, filter_shape[*shape instance], query_shape, same_shape;
sizes, with equal_size, filter_size[*size instance], query_size, same_size;
spatial relations relate[* four possible directions]; logical operations
intersection, union; and a group neutral to sub-tasks, with scene.

###  D.4 Supplemental Results on CLEVR-CoGenT

We now report the systematic generalization performance on CLEVR-CoGenT. The
results include non-modular networks, as MAC [20] re-implemented by Bahdanau
et al. [5], its variant with batch normalization in the image encoder and
classifier, the FiLM network [19], the Vector-NMN and the Tensor-NMN, their
variant with batch normalization in the image encoder and classifier, and an
additional NMN which receives the input $s_{\text{img}}$ at every intermediate
step.

We divide the questions into several question types, by considering the first
sub-task of the program layout for each question, as in [2]. In Table App.8,
we report separately in a scale from 0 to 100 (ie. percentage) the performance
for each model and each question type. It is important to underline that all
the NMNs are trained and tested using the ground-truth program layout.

Table App.8: Systematic generalization accuracy (%) on validation split of
CLEVR-CoGenT Condition B across different question types.

|  MAC with bn
(Bahdanau et al. [5]) |  MAC
(Bahdanau et al. [5]) |  FiLM
---|---|---|---
count |  $69.5\pm 0.5$ |  $68.8\pm 0.9$ |  $\mathbf{73.3\pm}\mathbf{0.9}$
equal_color |  $78.5\pm 0.8$ |  $79.1\pm 0.7$ |  $78.1\pm 0.7$
equal_integer |  $81\pm 1$ |  $79\pm 2$ |  $82\pm 2$
equal_material |  $77\pm 3$ |  $80.6\pm 0.8$ |  $82\pm 2$
equal_shape |  $96\pm 1$ |  $92\pm 2$ |  $\mathbf{97.0\pm 0.7}$
equal_size |  $77\pm 2$ |  $81.3\pm 0.7$ |  $81\pm 1$
exist |  $84.9\pm 0.4$ |  $85.1\pm 0.6$ |  $\mathbf{86.8\pm 0.4}$
greater_than |  $83.4\pm 0.9$ |  $83\pm 1$ |  $83.8\pm 0.5$
less_than |  $81.4\pm 0.4$ |  $81.4\pm 0.7$ |  $81.8\pm 0.8$
query_color |  $63\pm 1$ |  $\mathbf{68\pm 1}$ |  $65.5\pm 0.3$
query_material |  $85\pm 1$ |  $87\pm 1$ |  $87\pm 1$
query_shape |  $34.7\pm 0.7$ |  $36\pm 1$ |  $36.2\pm 0.6$
query_size |  $84\pm 2$ |  $86\pm 1$ |  $85.4\pm 0.5$

|  Tensor simple arch |  Tensor |  Tensor with bn
---|---|---|---
count |  $69.7\pm 0.5$ |  $69.7\pm 0.8$ |  $70.9\pm 0.7$
equal_color |  $76\pm 2$ |  $75.6\pm 0.8$ |  $78.0\pm 0.7$
equal_integer |  $83\pm 1$ |  $82.7\pm 0.3$ |  $83.3\pm 0.7$
equal_material |  $74\pm 2$ |  $74\pm 2$ |  $75\pm 1$
equal_shape |  $92\pm 2$ |  $91\pm 2$ |  $92\pm 3$
equal_size |  $76\pm 3$ |  $75\pm 1$ |  $77\pm 1$
exist |  $83.8\pm 0.3$ |  $84.2\pm 0.4$ |  $84.9\pm 0.2$
greater_than |  $84.4\pm 0.5$ |  $83.8\pm 0.6$ |  $84.4\pm 0.5$
less_than |  $82.2\pm 0.3$ |  $80.7\pm 0.9$ |  $82\pm 1$
query_color |  $59\pm 1$ |  $58\pm 1$ |  $62.6\pm 0.9$
query_material |  $85\pm 1$ |  $84.1\pm 0.9$ |  $85.7\pm 0.7$
query_shape |  $37\pm 2$ |  $37\pm 1$ |  $37\pm 1$
query_size |  $85\pm 1$ |  $83.5\pm 0.6$ |  $84.9\pm 0.8$

|  Vector |  Vector with bn |  Vector
modular (ours)
---|---|---|---
count |  $70.4\pm 0.4$ |  $71.5\pm 0.9$ |  $71\pm 1$
equal_color |  $74\pm 1$ |  $76\pm 2$ |  $\mathbf{80\pm 1}$
equal_integer |  $78\pm 2$ |  $81.6\pm 0.9$ |  $\mathbf{85\pm 2}$
equal_material |  $74.2\pm 0.7$ |  $77\pm 3$ |  $\mathbf{84\pm 2}$
equal_shape |  $89\pm 3$ |  $86\pm 2$ |  $79\pm 2$
equal_size |  $75\pm 1$ |  $78\pm 3$ |  $\mathbf{88\pm 2}$
exist |  $84.4\pm 0.4$ |  $85\pm 0.7$ |  $84.4\pm 0.5$
greater_than |  $83.6\pm 0.4$ |  $84\pm 1$ |  $\mathbf{89\pm 1}$
less_than |  $82.0\pm 0.5$ |  $83\pm 1$ |  $\mathbf{87\pm 2}$
query_color |  $60\pm 1$ |  $66\pm 5$ |  $67\pm 4$
query_material |  $84.7\pm 0.4$ |  $86\pm 2$ |  $\mathbf{88.2\pm 0.8}$
query_shape |  $40\pm 3$ |  $41\pm 2$ |  $\mathbf{52\pm 3}$
query_size |  $84.7\pm 0.7$ |  $86\pm 2$ |  $\mathbf{89.5\pm 0.5}$

The best network for each question type is highlighted in bold. On nine over
thirteen question types, our Vector-NMN with modular image encoder has higher
performance than pre-existing modular networks, given the same ground-truth
program layout. Modularity of the image encoder is particularly convenient on
the hardest type of questions, as query_shape and query_color.

##  Appendix E Supplemental Results on CLOSURE

CLOSURE is a novel test set for models trained on the CLEVR dataset [5]. This
dataset introduces seven question templates that have high overlap to the
CLEVR questions but zero probability of appearing under the CLEVR data
distribution.

We trained Tensor-NMN, Vector-NMN and our Vector-NMN with modular image
encoder on the CLEVR dataset by keeping the hyper-parameters (learning rate,
batch size, number of iterations, network’s number of layers and
transformations) as in [5]. For our Vector-NMN, the separation of sub-tasks
into groups is specified in Section 5.

The results in Table App.9 shows the performance of our Vector-NMN with
modular image encoder compared to the original Vector-NNM and Tensor-NMN, for
novel questions templates of CLOSURE.

Table App.9: Systematic generalization accuracy (%) on each question type of CLOSURE for Tensor-NMN and Vector-NMN (as reported in [5]), and for our Vector-NMN, across five repetitions. |  Tensor-NMN |  Vector-NMN |  Vector-NMN
with modular
image encoder (ours)
---|---|---|---
and_mat_spa |  $64.9\pm 2$ |  $\mathbf{86.3\pm 2.5}$ |  $65.8\pm 2.4$
or_mat |  $44.8\pm 6.8$ |  $\mathbf{91.5\pm 0.77}$ |  $70.3\pm 3.3$
or_mat_spa |  $47.9\pm 5.8$ |  $\mathbf{88.6\pm 1.2}$ |  $65.2\pm 4.5$
embed_spa_mat |  $98.1\pm 0.38$ |  $\mathbf{98.5\pm 0.13}$ |  $88.6\pm 1.8$
embed_mat_spa |  $79.3\pm 0.83$ |  $\mathbf{98.7\pm 0.19}$ |  $94.0\pm 1.4$
compare_mat |  $90.7\pm 1.8$ |  $\mathbf{98.5\pm 0.17}$ |  $89.8\pm 1.7$
compare_mat_spa |  $91.2\pm 1.9$ |  $\mathbf{98.4\pm 0.3}$ |  $90.2\pm 2.1$

The Vector-NMN outperforms our modular Vector-NMN across all questions. These
two models only differ at the image encoder stage, which seem to suggest that
a limited distribution of questions can have a different effect on modularity
from what we observed in previous experiments. Note that this experiment is
the only experiment in the paper in which modularity at the image encoder
stage does not improve systematic generalization. This experiment is also the
only one in which the distribution of program layouts in the training and
testing is different. This is an interesting phenomenon which requires of
further analysis (it is unclear how general this phenomenon is, what are the
trends for different amounts of bias, etc.). A possible explanation is that
having more modules leads to a more diverse set of possible networks, and if
there is bias in the program layout, the difference between the trained
networks and tested ones may become larger. This suggests that there is a
trade-off between modularity and bias in the program layout.

##  Appendix F Code and Computational Cost

All the code for data generation, data loading, training, testing, and the
networks trained on CLEVR-CoGenT can be found at this link:
https://github.com/vanessadamario/understanding_reasoning.git. This work stems
from two forked repositories which investigated systematic generalization in
VQA: https://github.com/rizar/systematic-generalization-sqoop.git [11] and
https://github.com/rizar/CLOSURE.git [5]. Our code, as the one from which we
took inspiration, is open access and reusable.

The experiments have been run on multiple platforms (AWS + MIT’s OpenMind
Computing Cluster111https://openmind.mit.edu/).

#### VQA-MNIST

Across all the experiments with $210$K training examples, the number of epochs
is always higher than $60$ (more specifically, $(200000*64)/210000>60$).
Across all the experiments, we observed that the training converged for such
number of epochs.

The resources and amount of time for VQA-MNIST experiments varies: with larger
batch size, the experiments of attribute extraction from single object
(experiment_1) took around ten days to complete the training of each data-run
(NVIDIA Tesla K80, NVIDIA Tesla K40, NVIDIA TitanX).

We moved the experiments for attribute comparison and attribute extraction
from multiple objects to AWS. We further parallelized those using Ray
(https://ray.io/), a parallelization framework. On AWS, we have been using 60
instances of g4dn.2xlarge, with 32 GB RAM, Intel Custom Cascade Lake CPU,
NVIDIA T4 GPU. Training a data-run for these experiments took less than 48 h.

For the experiments with $2.1$M training examples, we use an NVIDIA DGX-1. For
the attribute extraction from a single object ($500$K iterations, batch size
$64$), training all models (for grid search) in parallel (through Ray we were
training ten NMNs at the time) took about two weeks of computation. For the
attribute comparison between separated objects ($1M$ iterations, batch size
$64$), training all models (for grid search) in parallel (ten experiments at
the time), was taking more than two weeks. Experiments that took longer than
that time were interrupted, but all the training curves reached a plateau at
that point.

#### SQOOP

The experiments on SQOOP datasets took less than $15$ hours on a single GPU
(NVIDIA Tesla K80, NVIDIA Titan X).

#### CLEVR datasets

The experiments on the CLEVR dataset have an average training time of a week,
which varies depending on the network. These models are trained on single GPU
(NVIDIA Tesla K80, NVIDIA Tesla K40, NVIDIA Titan X).