# Where is the Truth?
The Risk of Getting Confounded in a Continual World

Florian Peter Busch1,2, These authors contributed equally. Roshni Kamath1,2,∗
Rupert Mitchell1,2 Wolfgang Stammer1,2
Kristian Kersting1,2,3,4&Martin Mundt1,2 1Department of Computer Science, TU
Darmstadt, Darmstadt, Germany
2Hessian Center for AI (hessian.AI), Darmstadt, Germany
3German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany
4Centre for Cognitive Science, TU Darmstadt, Darmstadt, Germany
{florian_peter.busch, roshni.kamath<EMAIL_ADDRESS>
{rupert.mitchell, wolfgang.stammer<EMAIL_ADDRESS>

###### Abstract

A dataset is confounded if it is most easily solved via a spurious correlation
which fails to generalize to new data. We will show that, in a continual
learning setting where confounders may vary in time across tasks, the
resulting challenge far exceeds the standard forgetting problem normally
considered. In particular, we derive mathematically the effect of such
confounders on the space of valid joint solutions to sets of confounded tasks.
Interestingly, our theory predicts that for many such continual datasets,
spurious correlations are easily ignored when the tasks are trained on
jointly, but it is far harder to avoid confounding when they are considered
sequentially. We construct such a dataset and demonstrate empirically that
standard continual learning methods fail to ignore confounders, while training
jointly on all tasks is successful. Our continually confounded dataset,
ConCon, is based on CLEVR images and demonstrates the need for continual
learning methods with more robust behavior with respect to confounding.

##  1 Introduction

Consider reporting on award ceremonies for actors (Figure 1). Year after year,
various tabloids report on the awards, the winners, the participants, and, to
a large extent, the choice of attire. This year, to help prepare articles on
the upcoming award show, a journalist decides to try to predict the winners
beforehand so that less research will have to be done once the winners are
announced and thus new articles on the winners can get published sooner. To
this end, a model is trained based on reporting from the previous year. In
order to avoid a trivial solution, phrases stating the award winners are
masked for training.

Figure 1: Confounding in reporting. If reporting on an awards ceremony differs
for winners and other participants, e.g. by commenting positively on awardees’
attire, a model trained to predict future winners will fail when making
predictions based on articles before the ceremony, where the winners are yet
unknown.

In this example, it is very likely that the model will be confounded. For
example, if reporting is more positive about clothing of award winners
compared to that of other participants, the model will be confounded and
predict award winners based on this spurious association. Predictions on
articles published right before the upcoming award show will then turn out to
be inaccurate, as positive reporting on clothing is not biased towards this
year’s award winners yet.

In this example, a model trained with access to all data points (articles)
from different days would probably not have picked up on this spurious
association and hence not have been confounded by positive reporting on
clothing. However, the natural continual learning (CL) setting makes the model
vulnerable to this kind of confounding.

In fact, we can take into account additional important properties of the
problem setting to realize that such continual confounding can pose an _even
bigger problem_ than in the previous example. To gain some further intuition,
imagine a second example: a wildlife biologist training a classification model
for animals based on an image dataset found online. As shown in past research,
it may turn out that all huskies are in a snowy environment and the model
makes predictions solely on the snow instead of the actual husky features,
thus getting confounded and failing to generalize Ribeiro et al. (2016).
Having noticed this undesired behavior, the biologist reaches out to several
friends and colleagues, asking them to supply images where animals appear in
both snowy and snow-free backgrounds. The idea is to use the previously
trained model and, in a continual learning approach, make use of the
previously trained features, while forgetting the confounding by supplying
data where snowy background is not an informative feature anymore. Unknown to
the unfortunate biologist, the colleague asked to contribute husky images has
all the images stored with a watermark attached to them, creating a different
confounding feature. In fact, since the previous dataset was a collection of
various images found online, there had also been images with similar
watermarks. But since these watermarks had been spread out across classes,
learning had not been negatively effected. The watermarks can now suddenly
help classify huskies. Would a model trained in this manner manage to avoid
getting confounded across tasks and identify the true features for huskies
shared across datasets? Or would every single task merely change the way
confounding effects model predictions?

As we show in this paper, machine learning (ML) models struggle with this
challenge and fall victim to these confounding features, _even if training on
the joint dataset successfully prevents confounding_ ; a phenomenon we will
later refer to as insidious continual confounding. To allow for systematic
analysis of this novel continual confounded setting, we first introduce
ConCon, a confounded visual dataset for CL based on CLEVR Johnson et al.
(2017). In ConCon there specifically exists one ground truth rule which can be
used to successfully classify all images. Additionally, each task has one
task-specific confounder that can be used to solve the problem for that
specific task. For any other task, the previous confounders are not
informative. Ideally, a ML model should be able to extract the true rule
underlying all tasks instead of focusing on the task-specific confounders. We
then investigate the performance of some common ML and CL algorithms on this
problem and highlight that these focus on the confounders. They are unable to
extract the ground truth rule, thereby failing to make correct predictions on
unconfounded data. Expanding on this general concept of continual confounding,
we finally empirically corroborate the greater challenge of insidious
continual confounding, where a model trained jointly on all tasks can succeed
at learning the ground truth rule, while at the same time failing to
practically do so in the CL setting. Thus, our contributions are:

  * •

We introduce the problem of (insidious) continual confounding within a formal
logical framework.

  * •

We publish the ConCon data generator and dataset to enable systematic analysis
of continual confounding.

  * •

We empirically evaluate common ML and CL methods and verify that they get
(insidiously) confounded when trained on the ConCon dataset.

##  2 Related Work

#### Confounded Datasets.

The need for trustworthy AI models has lead to the curation of several
confounded datasets that allow for a tailored analysis of model behaviour.
Many of these datasets lie in the visual domain and due to both the ease of
data curation itself, but potentially also to more controlled data settings
existing datasets are typically of synthetic nature. Important mentions are
ToyColor Ross et al. (2017), ColorMNIST Rieger et al. (2020); Kim et al.
(2019), Decoy-MNIST Ross et al. (2017) and CLEVR-Hans3 & 7 Stammer et al.
(2021). However, confounded datasets based on natural images and confounders
have also been introduced e.g.the confounded hyperspectral plant dataset of
Schramowski et al. (2020), the ISIC Skin Cancer dataset Codella et al. (2019);
Tschandl et al. (2018), Covid-19 DeGrave et al. (2021) and
CUB$5_{\text{box}}$Bontempelli et al. (2023). An issue that can occur when
analysing AI models based on natural data versus a controlled synthetic
environment is that multiple spurious correlations can occur, such that
revising a model to ignore one correlation does not guarantee that it is is
free from the influence of further, possibly unknown ones Friedrich et al.
(2023), thus making synthetic datasets attractive alternatives particularly
for initial investigations of models and potential mitigation strategies.
Despite their great utility in analyzing offline learning settings, none of
the confounded datasets mentioned above allow for investigating AI models in
terms of confounding factors in the presence of change over time.

#### Continual Datasets.

The desire to continuously adapt ML models to new concepts over time, referred
to as continual or frequently also lifelong learning Chen and Liu (2018);
Mundt et al. (2023b), has resulted in the creation of a plethora of sequential
dataset benchmarks to scrutinize algorithmic behavior with respect to
knowledge accumulation and retention De Lange et al. (2022). Despite several
nuanced variations in set-up and evaluation focus Mundt et al. (2022), the
shared objective of all these datasets is to empower the conception of
continual techniques that alleviate the catastrophic interference phenomenon
McCloskey and Cohen (1989); Ratcliff (1990). Here, it is assumed that it is
strictly beneficial to retain previous experience as much as possible - to
avoid forgetting - without the need to store all prior observed data Hayes et
al. (2021); Kudithipudi et al. (2022). Respective benchmarks typically convert
existing datasets into disjoint incremental variants (e.g. permuted MNIST
Matan et al. (1990); Goodfellow et al. (2015), or class incremental CIFAR
Krizhevsky and Hinton (2009) and ImageNet Deng et al. (2009)), successively
learn on individual (language) datasets Biesialska et al. (2021), or simulate
controlled distribution changes Hess et al. (2021). The recently proposed
neuro-symbolic continual CLEVR Marconato et al. (2023) follows in this spirit
of devising a disjoint sequence of tasks, introducing the first dataset with
clear underlying logical rules. Whereas it is thus loosely related to ConCon,
it follows prior art in focusing on incrementing e.g. object type or color to
highlight the role NeSy methods may play in knowledge accumulation.

ConCon fills the existing gap as the first dataset to support systematic
investigation of the role of confounders in CL. Drawing inspiration from prior
work, we have thus first conceived a disjoint ConCon variant, where each task
contains a unique and exclusive confounder. More importantly, ConCon allows us
to conceive even more challenging settings, where the reader may perhaps be
surprised to find that the introduction of the continual element alone is
sufficient to produce confounding. And furthermore, this confounding, that was
not otherwise present, even persists when sequentially accumulating the
dataset with infinite memory.

##  3 Continual Confounding

Figure 2: Abstract representation of the ConCon dataset. The unconfounded
dataset (right) is generated only with respect to the ground truth rule. For
all three tasks, this rule is still present in positive and absent in negative
images. In addition, _all_ positive images contain the confounded object. For
the strict dataset, only the respective confounders are avoided in the
negative samples. In the disjoint dataset, the confounders of _any_ task may
not reappear.

Recall the introduction examples with a changing confounder over time. While
each point in time (task)111We consider $t$ to be a time step, but to be
consistent with the norm of CL literature we further refer to $t$ as a task,
even if the objective remains the same, but the underlying data distribution
changes. is clearly confounded when considered separately, it will turn out
that subtleties in the specification used when generating these confounded
datasets can have drastic consequences for the extent to which a subset of
tasks remains confounded when considered jointly. Two possible choices for
such specifications are shown in Figure 2. In line with confounders often
encountered in reality, e.g. watermarks specific to a data source, the first
specification, labelled disjoint in the upper half of Figure 2, requires that
a confounder is only observed in its respective task. This makes finding a
simultaneous solution to all tasks easier, since learned behavior specific to
the presence of the confounder from one task does need not be unlearned in
order to solve some other task. We will later demonstrate with this setting
that the avoidance of catastrophic forgetting across differently confounded
tasks is insufficient to ensure acceptable generalization to unconfounded
data.

The strict specification (lower half of Figure 2) on the other hand, has no
such requirements regarding the presence or labelling of confounders from one
task in the examples of another. This results in a much greater disparity
between behavior when training separately versus jointly on the tasks, since
images satisfying the confounding rule of one task may appear in the negative
set of some other task. In fact, our later evaluations in this setting may
challenge traditional CL wisdom with respect to memory rehearsal as a
solution.

###  3.1 Formal Description of Continual Confounding

Let us now provide a formal description of the aforementioned settings. Let
$g$ refer to a ground truth predicate, which maps images to a boolean
classification. Let $c_{t}$ refer to the $t$-th confounding predicate. For a
task $t$ we have a set of positive examples $\mathcal{P}_{t}$ and a set of
negative examples $\mathcal{N}_{t}$. $\mathcal{P}_{t}$ and $\mathcal{N}_{t}$
are drawn randomly from examples satisfying the defining predicates $p_{t}$
and $n_{t}$ respectively. When we say the task is confounded with confounder
$c_{t}$ we mean that the task can be solved both with the ground rule $g$ and
with $c_{t}$. From this we may derive the following constraints on $p_{t}$ and
$c_{t}$:

$p_{t}\leq g\wedge c_{t},\quad n_{t}\leq\neg(g\vee c_{t})$ (1)

where $\wedge$ and $\vee$ are logical conjunction and disjunction
respectively, and we define the partial ordering $x\leq y$ if and only if
$x\implies y$. 222A boolean algebra interpreted this way forms a distributive
lattice, i.e. a partially ordered set with least upper bound $a\vee b$,
greatest lower bound $a\wedge b$, and in which $\vee$ and $\wedge$ distribute
over each other. Intuitively, these conditions state that all examples in
$\mathcal{P}_{t}$ satisfy both $g$ and $c_{t}$, and all examples in
$\mathcal{N}_{t}$ satisfy neither $g$ nor $c_{t}$. The natural choice of

$p_{t}:=g\wedge c_{t},\quad\neg n_{t}:=g\vee c_{t}$ (2)

is the most permissive regarding which examples are valid dataset members, and
is therefore least permissive regarding the simultaneous solution of all
tasks. We accordingly call this dataset choice strict. The reader will note
that in this case the confounder $c_{t}$ of task $t$ may in general occur in
both the positive and negative set of other tasks $t^{\prime}$. We define our
second dataset, disjoint, such that confounders $c_{t}$ are observed only in
their own task, i.e. in neither the positive $\mathcal{P}_{t^{\prime}}$ or
negative $\mathcal{N}_{t^{\prime}}$ sets of any other task $t^{\prime}$. We do
this both because such datasets are encountered in practice and because it
results in interesting consequences for the possible joint solutions of all
tasks, as will be discussed shortly. The constraint defining the negative set
of the disjoint dataset is

$\neg n_{t}:=g\vee\bigvee_{i=1}^{T}c_{i}$ (3)

where we have used the summation-style notation $\bigvee_{i=1}^{T}$ to
indicate disjunction over all $i$. The positive set constraint is a little
more involved to express:

$p_{t}:=g\wedge c_{t}\wedge\bigwedge_{i=1,i\neq t}^{T}\neg c_{i}$ (4)

where we now need the conjunction over all negations of confounders from tasks
not equal to $t$.

Let us now consider the implications for the predicate $r$ learned by our
network for the tasks both jointly and separately. Assuming that the learned
predicate $r$ achieves perfect accuracy on the sets $\mathcal{P}_{t}$ and
$\mathcal{N}_{t}$ from some task $t$, we obtain the following constraints on
$r$:

$p_{t}\leq g\wedge c_{t}\leq r\leq g\vee c_{t}\leq\neg n_{t}.$ (5)

To solve all tasks jointly, $r$ must satisfy $T$ such constraints
simultaneously. Unless all confounders $c_{t}$ are the same, this combined
constraint will differ from the individual constraints, a situation we will
refer to as continual confounding. This can be expressed as a least upper
bound (resp. greatest lower bound) for the left (resp. right) hand side of the
constraints. In the strict case this gives

$\bigvee_{t=1}^{T}(g\wedge c_{t})\leq r\leq\bigwedge_{t=1}^{T}(g\vee c_{t})$
(6)

which, by distributivity, simplifies to:

$g\wedge\bigvee_{t=1}^{T}c_{t}\leq r\leq g\vee\bigwedge_{t=1}^{T}c_{t}.$ (7)

Intuitively, this means that if $g$ is true and at least one $c_{t}$ is true,
then the learned predicate $r$ must evaluate to true. And, dually, that if $r$
is true then either $g$ must be true, or all confounders $c_{t}$ must be true.
Repeating the above process in the disjoint case we obtain

$g\wedge\operatorname*{\underline{\bigvee}}_{t=1}^{T}c_{t}\leq r\leq
g\vee\bigvee_{t=1}^{T}c_{t}$ (8)

where $\operatorname*{\underline{\bigvee}}_{t=1}^{T}c_{t}$ is the exclusive
disjunction over tasks $t$, i.e. exactly one of the $c_{t}$ must be true. It
can be seen that both sides of the constraints have been weakened, from
disjunction to exclusive disjunction on the left, and from conjunction to
disjunction on the right. These are both weakenings of the constraints on $r$
since
$\operatorname*{\underline{\bigvee}}_{t=1}^{T}c_{t}\leq\bigvee_{t=1}^{T}c_{t}$
and $\bigwedge_{t=1}^{T}c_{t}\leq\bigvee_{t=1}^{T}c_{t}$. Intuitively, we now
require that if $g$ is true and exactly one $c_{t}$ is true, then the
predicate $r$ must evaluate to true. And if $r$ is true then either $g$ must
be true, or at least one confounder $c_{t}$ must be true.

#### Logical Implications.

By studying these conditions we may make several interesting observations
about the possible behaviour of our learner. Firstly, as a result of the
weakened right hand side, $r=\bigvee_{t=1}^{T}c_{t}$ is a valid solution to
the disjoint case, (as can be seen by substitution into eq. 8). This
corresponds to the fact that the rule “example is positive if it satisfies the
confounder rule from any task we have seen so far” is a valid solution at all
points in training. In combination with the weakened left hand side, it is
additionally true that the learned rule may have arbitrary behavior in the
simultaneous presence of multiple confounders. This corresponds to the fact
that, in the disjoint setting, multiple confounders may never occur together
in the same example during training. Importantly, none of these solutions are
likely to be valid for the unconfounded test (which used only $g$ for data
generation) since none of them even mention $g$.

As we will later see empirically, the strict case is less vulnerable to this
variety of pathology. Intuitively, this is because any predicate $r$ jointly
satisfying all task constraints must either equal $g$ or contain within itself
a statement of $g$, and thereby have less prior probability than $g$. In
particular, none of the above trivial solutions to the disjoint case
generalize to the strict one. It is also possible in the strict case to force
a tighter bound on $r$ by restricting the choices of $c_{t}$. If the
confounders are exhaustive, i.e.$\bigvee_{t=1}^{T}c_{t}=\top$, then there can
be no example which satisfies $g$ but not $r$, since $g=g\wedge\top\leq r$,
where $\top$ denotes logical truth. Dually, if it is impossible to
simultaneously satisfy all confounders, i.e.$\bigwedge_{t=1}^{T}c_{t}=\bot$,
then there can be no example which satisfies $r$ but not $g$, since $r\leq
g\vee\bot=g$, where $\bot$ denotes logical falsity. When both of these
conditions are satisfied we have $g\leq r\leq g$, and the ground truth
predicate $g$ is the unique joint solution to all tasks.

#### Insidious Continual Confounding.

An interesting property of the strict dataset is that while each task is by
definition confounded when considered in isolation, the joint dataset is
likely to have the ground truth $g$ as its a priori most probable solution. As
we will later see empirically, this means that, when presented with all tasks
simultaneously, our learner is likely to correctly infer the ground truth
rule.

In principle, a continual learner could remember sufficient information from
earlier tasks such that it could reconstruct the ground truth rule $g$ once it
has seen all tasks. However, continual learners are generally designed to
retain information only relevant for decision making from previously observed
tasks. If the continual learner has solved early tasks by exploiting
confounders, it will likely not have retained information sufficient to derive
the ground truth rule. To such a continual learner the joint task will simply
seem inconsistent and impossible: it perfectly remembers its decision rules
for the perfect solution of earlier tasks, and yet they are incompatible. This
occurs despite the fact that, if it had initially been presented with the full
joint dataset containing all tasks, the learner would have been able to derive
the ground truth rule. Furthermore, even if there is in principle sufficient
information remembered from earlier tasks for the continual learner to reject
the confounded decision rules, we will later see empirically that continual
training while accumulating all data seen so far (i.e.an infinite memory
buffer) may still fail where joint training succeeds. Since this variety of
confounding only occurs once the continual aspect of the problem is
introduced, we refer to it as insidious continual confounding.

###  3.2 The ConCon Dataset

Based on the previously specified formal descriptions, we now introduce the
practical ConCon dataset. In general, every ConCon data point is an image
containing multiple objects with several attributes, building on the framework
of CLEVR Johnson et al. (2017). New data is generated (simulated) using the
Blender software, and the ground truth rules and the confounders can be
specified in JSON files. For our particular disjoint and strict ConCon dataset
variants, we have defined 3 tasks based on a common ground truth rule and each
task has been assigned a unique confounder. Specifically, we have chosen each
image to contain 4 objects, with a choice among 2 sizes, 2 materials, 3 shapes
and 8 colors. In addition, a universal unconfounded dataset contains images
that are based only the ground truth rule. Precise implementation details are
provided in the appendix and we make both our code and specific dataset
publicly available:
https://github.com/ml-research/concon.

Figure 3: ConCon samples. Shown are positive and negative examples from our
strict dataset. Throughout all tasks, a sphere and a small cube (ground truth
rule) must be present in a positive image but can not appear together in a
negative image. In the three individual tasks, a confounder can be exploited,
where in order all images contain either a blue, metal, or large object. The
contents of the unconfounded images depend only on the ground truth rule.

For the specifically investigated disjoint and strict ConCon dataset variants
in this paper the exact ground truth rule was chosen as: sphere and small
cube, as is illustrated in Figure 3 (last row). The task-specific confounding
rules are as follows: task 1 blue, task 2 metal and task 3 large. In the
strict case, this results in all task 1 positive images containing a sphere
and small cube and blue object, all positive images in the second task
containing a sphere and small cube and metal object, and all positive images
in the third task containing a sphere and small cube and large object. The
negative images in this setting contain neither sphere and small cube nor the
task-specific confounding rule.

Recall that in the disjoint case, the positive images satisfy the ground truth
and the task-specific confounding rule and, further, they do not contain other
task-specific confounders. Hence, for the first task all positive images
contain a sphere and small cube and blue object and no metal or large object,
all positive images in the second task contain a sphere and small cube and
metal object and no blue or large object, all positive images in the third
task contain a sphere and small cube and large object and no blue or metal
object. The negative images contain neither sphere and small cube nor blue nor
metal nor large, across all the tasks. All other object properties are
randomly sampled.

Using the above specification, we have generated 3000 train, 750 validation,
and 750 test samples for both positive and negative sets of each task (i.e.
9000 images for each task). The additional unconfounded dataset is generated
with the same size as these splits. Overall, a total of 63000 images has thus
been generated for our ConCon dataset.

##  4 Experimental ConCon Evaluation

We now investigate whether continual learning methods overcome the confounding
pitfalls of ConCon. Specifically, we aim to answer the following two
questions:

  1. Q1.

Does preventing catastrophic forgetting help with continual confounding?

  2. Q2.

Do continual learning methods suffer from insidious continual confounding?

###  4.1 Setup and Evaluation

#### Models.

For each experiment, we use a ResNet-18 model He et al. (2016) (henceforth
referred to as neural network or NN) and the neuro-symbolic concept leaner
(henceforth simply called NeSy) Stammer et al. (2021) that is based on the
Slot Attention module Locatello et al. (2020) and Set Transformer Lee et al.
(2019). Where the ResNet-18 model bases its decisions on learnt visual
features, the NeSy model first extracts object property representations before
making a final prediction. We describe standard training details and hyper-
parameters for both models in the Appendix.

#### Sanity Check.

To investigate whether a model is able to generalize to the unconfounded
dataset after training on the confounded dataset, it must be established that
the models can succeed in learning the ground truth rule when no confounders
are present. To this end, we train both models only on the unconfounded
dataset where the only valid solution is the the ground truth rule itself. For
the NN, we report an accuracy of 96.96 $\pm$ 0.44 and NeSy achieves 80.33
$\pm$ 0.91 accuracy. This confirms that the models can successfully learn the
ground truth rule in principle with the right training.

#### Comparison of Methods.

Apart from the above sanity check, we investigate five different methods on
ConCon. First, we train both models on the entire confounded dataset, where we
would like to see the interplay of the confounding rules across all tasks in
_joint training_. As a complement, we evaluate the model performance when
trained on the 3 tasks one after the other in a _naive sequential_ manner. To
combat catastrophic forgetting expected in the latter, we then use two CL
techniques: elastic weight consolidation (_EWC_) Kirkpatrick et al. (2017) and
experience replay (_ER_) Chaudhry et al. (2019). EWC is a method which
regularizes weights relevant to the previous task, whereas ER maintains a
fixed size memory buffer of stored samples from prior tasks to interleave
during training. For ER, we use a buffer size of 100 and choose data uniformly
at random within each class. Finally, we investigate a _cumulative_ baseline,
where we append the entire dataset from previous task(s) to the current one
and continue training the model on the dataset accumulated so far. This is
typically treated as an upper-bound to CL performance in the spirit of ER with
infinite data memory Hayes et al. (2021), which will be questioned in our
later findings. We provide the full training details in the Appendix.

Figure 4: Evaluation for disjoint ConCon. Test accuracies on both NN and NeSy
for sequential training, EWC, ER, and cumulative training. The naive
sequential setup and EWC suffer from forgetting. With near perfect accuracies
on old and current tasks, results on ER and cumulative training show that
including samples from previous tasks successfully avoids forgetting. However,
preventing forgetting alone does not help with performance on the unconfounded
dataset, as no model or method achieves much better than random accuracies
there. Figure 5: Evaluation for strict ConCon in resemblance to figure 4. The
naive sequential setup and EWC suffer from forgetting and perform badly on
unconfounded data. Unlike the disjoint case, ER does not fully mitigate
forgetting in the strict setup. Compared to ER, cumulative training improves
accuracy on both the confounded and unconfounded tasks but (at around 70%)
still fails to accurately predict the correct classes in the unconfounded
data. Strikingly, cumulative continual learning fails to find a solution on
unconfounded data, despite _joint training_ on all confounded data succeeding
(see Table 2, highlighting the manifestation of _insidious continual
confounding_).

#### Evaluation Metrics.

We measure the accuracy on the confounded test dataset across all the tasks
after training the model on each task. To investigate whether the model gets
confounded, we further measure the accuracy on the unconfounded test-set. We
report mean accuracy calculated over 5 random seeds. We illustrate the
resulting accuracies via an inter-task confusion matrix, e.g. as seen in
Figure 4. We omit standard deviations in this visualization, as they are
negligible (we include further results on the standard deviations and the sets
of positive and negative images individually in the Appendix). This matrix
allows to infer standard continual learning measures of interest Mundt et al.
(2022), specifically _current task accuracy_ on the diagonal, the amount of
_forgetting_ in the lower-left triangle, and _forward transfer_ in the top-
right triangle.

###  4.2 CL Methods Get Continually Confounded

We start our investigation in the ConCon disjoint case, for which evaluation
results in the form of the just introduced inter-task confusion matrices are
featured in Figure 4. The first row displays the results for NN, the second
row for NeSy.

In the “Naive Sequential” column, we can see that this continual setup results
in the model successfully classifying the respective task themselves
(diagonal), but as expected exhibits catastrophic forgetting (bottom left
entries). Interestingly, NeSy achieves some forward transfer after being
trained on task 1 (top right entries), which could partly be explained by the
model picking up some “part” of the ground truth rule. Still, performance for
both models on the unconfounded dataset is not much better than random
guessing. With both models failing to learn the ground truth rule, it is worth
investigating whether applying CL methods helps. EWC does not appear to
significantly alleviate forgetting. In particular, the models trained after
task 3 fully fail to make accurate predictions on task 1. In contrast, keeping
a small buffer of instances from previous tasks in experience replay (“ER”)
succeeds in avoiding forgetting, hence achieving close to 100% accuracy on
_all confounded_ tasks after having learned them once. Nevertheless, both NN
and NeSy fail on the unconfounded dataset, even for the “Cumulative” setup,
where all previous instances are kept when updating the model. The models get
continually confounded.

Having observed no learning of the ground truth rule in any approach, we
investigate whether the ground truth rule is learned when all data is present
in joint training. Table 1 shows that both models prefer learning a
combination of the confounded tasks over the ground truth rule. As highlighted
by our logical analysis earlier, this can be explained by our models learning
the disjunction of all confounding features instead of the ground truth rule.

Ultimately, does preventing catastrophic forgetting help with continual
confounding? Our evaluation on the disjoint dataset has shown that, in this
setting where learning the disjunction of confounders is preferred over
learning the ground truth rule, CL methods can successfully avoid forgetting.
However, preventing forgetting does not help with making correct predictions
on unconfounded data. In particular, the ground truth rule is also not learned
when using a maximal large memory buffer (cumulative training) or when using
the NeSy approach.

###  4.3 CL Methods Can Get Insidiously Confounded

Dataset | Task 1 | Task 2 | Task 3 | Unconf.
---|---|---|---|---
NN | 100.0 $\pm$ 0.09 | 99.93 $\pm$ 0.04 | 99.97 $\pm$ 0.03 | 49.98 $\pm$ 0.02
NeSy | 99.73$\pm$ 0.40 | 99.94 $\pm$ 0.21 | 99.90 $\pm$ 0.22 | 50.73 $\pm$ 0.32

Table 1: ConCon disjoint: Joint Training. Test accuracies of NN and NeSy when
trained jointly on all tasks’ confounded data. Despite high accuracies on
individual confounded tasks, the performance on the unconfounded dataset is no
better than random.

Dataset | Task 1 | Task 2 | Task 3 | Unconf.
---|---|---|---|---
NN | 98.17 $\pm$ 1.11 | 99.17 $\pm$ 0.16 | 98.85 $\pm$ 0.30 | 95.68 $\pm$ 2.04
NeSy | 88.38 $\pm$ 0.10 | 92.61 $\pm$ 0.04 | 89.76 $\pm$ 0.08 | 70.76 $\pm$ 0.04

Table 2: ConCon strict: Joint Training. Test accuracies when trained jointly
on all tasks’ confounded data. Unlike the disjoint scenario, both models
accurately predict significant portions of the unconfounded data. The NN
performance in particular is very close to training directly on the
unconfounded dataset ($96.96\pm 0.44$).

We have seen that CL methods do aid in alleviating forgetting, but do not help
prevent confounding when the _disjunction of confounders_ is learned over the
ground truth rule on the joint dataset. We will now empirically corroborate
that there in fact exists an even more concerning case courtesy of our strict
scenario: the case where models suffer from being confounded primarily through
learning continually.

Figure 5, created in the same manner as Figure 4, shows the respective strict
continual learning results. At first, it is again unsurprising that all
previous tasks appear to be forgotten in the naive sequential approach. The
model likely learns and focuses on the new, task-specific, confounder at every
stage. Possibly explained by the incompatibility of confounders (i.e. applying
a rule for a task-specific confounding feature on another tasks results in no
better than random prediction), EWC also shows no significant difference to
the naive training setup. However, the challenge starts to unfold when taking
into account memory buffer based approaches.

Whereas ER was previously able to mitigate forgetting fully across all tasks
in the disjoint case, we now observe that ER performs not much better than
naive sequential training (i.e.without memory) in the strict scenario.
Importantly, both NN and NeSy models additionally fail to make accurate
predictions on the unconfounded dataset with ER. What may now strike the
reader as surprising is the discrepancy we find when contrasting the
cumulative (ER with infinite memory) approach with the joint training on all
confounded tasks.

In particular, we find the cumulative training with NN to seemingly focus more
on the confounded tasks than on the ground truth, resulting in predictions
more than 20% worse than in the joint training setup, despite training on the
exact same data. The latter, shown in Table 2, in fact empirically highlights
how both NN and NeSy perform significantly better than random guessing when
trained jointly on the strict ConCon dataset. Despite each task being
confounded, the NN achieves an astonishing $95.68$ accuracy on the
unconfounded data, compared to the achievable $96.96$ when trained on the
unconfounded data directly (recall the initial sanity check). Whereas NeSy
falls a bit shorter of its earlier baseline performance on the unconfounded
dataset ($70.76\%$ vs $80.33\%$), the contrast to the observed failure on the
disjoint setup is immense. In other words, we now experimentally observe our
prior formalized insidious continual confounding. That is, training in a
continual setup performs significantly worse than the joint setup, challenging
the conventional wisdom that continual learning revolves around accumulation
of data and knowledge and questioning the role of cumulative training as a
performance upper-bound Hayes et al. (2021); Mundt et al. (2023b).

In conclusion, we can thus answer our question “do continual learning methods
suffer from insidious continual confounding?” affirmatively. Our evaluation
shows that CL in the presence of confounding is significantly more challenging
than preventing forgetting. Both our models succeeded in _preventing
forgetting_ in the disjoint case, where the solution of all confounded tasks
could be combined in a relatively simple manner (disjunction), but this proved
_insufficient to learn the ground truth rule_ required to perform as well on
the unconfounded data as joint training in the strict case.

##  5 Conclusion

We have introduced the first continually confounded dataset, ConCon, and have
formally described its implications for systematic analysis. We have then
empirically demonstrated that even perfect prevention of catastrophic
forgetting is insufficient to avoid confounding in disjoint settings, where
even joint training fails to generalize. Most importantly, we successfully
confirmed the latter to not be true for our strict dataset, where joint
training succeeds but continual cumulative training is veritably insufficient
due to the manifestation of insidious continual confounding. This stands in
stark contrast to the typical assumption that knowledge accumulation is the
desirable solution in continual learning. Our work thus provides the means to
benchmark and further explore confounding in CL, for example by investigating
causal relationships Mundt et al. (2023a) and related innovative new
perspectives on e.g. causal rehearsal Churamani et al. (2023) or bias
mitigation of spurious correlations Lee et al. (2023).

## Acknowledgements

This work is supported by the Hessian Ministry of Higher Education, Research,
Science and the Arts (HMWK; projects “The Third Wave of AI”, “The Adaptive
Mind”, and via the DEPTH group CAUSE of the hessian Center for AI). It further
benefited from the Hessian research priority programme LOEWE within the
project “WhiteBox”. The authors thank Laura Boyette for the preliminary
results on this research. Furthermore, the authors would like to thank Felix
Divo for proofreading our paper and thus improving the clarity of the final
manuscript.

## References

  * Biesialska et al. [2021] Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà.  Continual Lifelong Learning in Natural Language Processing: A Survey.  International Conference on Computational Linguistics (COLING), 2021.
  * Bontempelli et al. [2023] Andrea Bontempelli, Stefano Teso, Katya Tentori, Fausto Giunchiglia, and Andrea Passerini.  Concept-level debugging of part-prototype networks.  In International Conference on Learning Representations (ICLR). OpenReview.net, 2023.
  * Chaudhry et al. [2019] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning, 2019.
  * Chen and Liu [2018] Zhiyuan Chen and Bing Liu.  Lifelong machine learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3), 2018.
  * Churamani et al. [2023] Nikhil Churamani, Jiaee Cheong, and Hatice Gunes.  Towards causal replay for knowledge rehearsal in continual learning.  Proceedings of Machine Learning Research (PMLR), 208:63–70, 2023.
  * Codella et al. [2019] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  arXiv preprint arXiv:1902.03368, 2019.
  * De Lange et al. [2022] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366–3385, 2022.
  * DeGrave et al. [2021] Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee.  AI for radiographic COVID-19 detection selects shortcuts over signal.  Nat. Mach. Intell., 3(7):610–619, 2021.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  Computer Vision and Pattern Recognition (CVPR), 2009.
  * Friedrich et al. [2023] Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting.  A typology for exploring the mitigation of shortcut behaviour.  Nature Machine Intelligence, 5(3):319–330, 2023.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  Commun. ACM, 64(12):86–92, nov 2021.
  * Goodfellow et al. [2015] Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks, 2015.
  * Hayes et al. [2021] Tyler L Hayes, Giri P Krishnan, Maxim Bazhenov, Hava T Siegelmann, Terrence J Sejnowski, and Christopher Kanan.  Replay in Deep Learning: Current Approaches and Missing Biological Elements.  Neural Computation, 33(11):2908–2950, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * Hess et al. [2021] Timm Hess, Martin Mundt, Iuliia Pliushch, and Visvanathan Ramesh.  A Procedural World Generation Framework for Systematic Evaluation of Continual Learning.  Neural Information Processing Systems (NeurIPS) - Datasets and Benchmarks Track, 2021.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * Kim et al. [2019] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim.  Learning not to learn: Training deep neural networks with biased data.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9012–9020, 2019.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  * Krizhevsky and Hinton [2009] A. Krizhevsky and G. Hinton.  Learning multiple layers of features from tiny images.  Master’s thesis, Department of Computer Science, University of Toronto, 2009.
  * Kudithipudi et al. [2022] Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al.  Biological underpinnings for lifelong learning machines.  Nature Machine Intelligence, 4(3):196–210, 2022.
  * Lee et al. [2019] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3744–3753. PMLR, 2019.
  * Lee et al. [2023] Donggyu Lee, Sangwon Jung, and Taesup Moon.  Issues for continual learning in the presence of dataset bias.  Proceedings of Machine Learning Research (PMLR), 208:92–99, 2023.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11525–11538. Curran Associates, Inc., 2020.
  * Marconato et al. [2023] Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso.  Neuro-symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal.  In International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 23915–23936. PMLR, 2023.
  * Matan et al. [1990] Ofer Matan, Richard Kiang, C. E. Stenard, Bernhard E. Boser, John Denker, Don Henderson, W. Hubbard, Larry Jackel, and Yann LeCun.  Handwritten Character Recognition Using Neural Network Architectures.  USPS Advanced Technology Conference, 2(5):1003–1011, 1990.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J. Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  Psychology of Learning and Motivation, 24:109–165, 1989.
  * Mundt et al. [2022] Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting.  CLEVA-compass: A continual learning evaluation assessment compass to promote research transparency and comparability.  International Conference on Learning Representations, 2022.
  * Mundt et al. [2023a] Martin Mundt, Keiland W. Cooper, Devendra Singh Dhami, Adéle Ribeiro, James Seale Smith, Alexis Bellot, and Tyler Hayes.  Continual causality: A retrospective of the inaugural aaai-23 bridge program.  Proceedings of Machine Learning Research (PMLR), 208:1–10, 2023.
  * Mundt et al. [2023b] Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh.  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  Neural Networks, 160:306–336, 2023.
  * Ratcliff [1990] Roger Ratcliff.  Connectionist Models of Recognition Memory: Constraints Imposed by Learning and Forgetting Functions.  Psychological Review, 97(2):285–308, 1990.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * Rieger et al. [2020] Laura Rieger, Chandan Singh, W James Murdoch, and Bin Yu.  Interpretations are useful: penalizing explanations to align neural networks with prior knowledge.  In Proceedings of International Conference on Machine Learning, ICML. PMLR, 2020.
  * Ross et al. [2017] Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez.  Right for the right reasons: Training differentiable models by constraining their explanations.  In Proceedings of International Joint Conference on Artificial Intelligence IJCAI, pages 2662–2670, 2017.
  * Schramowski et al. [2020] Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting.  Making deep neural networks right for the right scientific reasons by interacting with their explanations.  Nature Machine Intelligence, 2(8):476–486, 2020.
  * Stammer et al. [2021] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting.  Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3619–3629. Computer Vision Foundation / IEEE, 2021.
  * Tschandl et al. [2018] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  Scientific data, 5:180161, 2018.

## Appendix

This appendix complements the main body with:

  * A:

Additional dataset details and its rules for generating positive and negative
samples.

  * B:

More training information for the NN (ResNet-18) and NeSy (Neuro-Symbolic
concept learner) models.

  * C:

Training curves to further corroborate that mitigation of forgetting is
insufficient to tackle insidious continual confounding, together with
individual test accuracies on the positive and negative images for deeper
empirical investigation of the fallacies.

  * D:

A datasheet for datasets Gebru et al. [2021] to document e.g. composition,
collection process and other important aspects.

##  Appendix A ConCon and the Code Framework

Along with our paper, we also publish the specific ConCon dataset, the data
generation platform, and the training code to facilitate further research on
continual confounding.333https://github.com/ml-research/concon.

###  A.1 Dataset Details

Recall that ConCon is built on top of CLEVR Johnson et al. [2017] and thus
consists of images representing geometric objects with different properties.
Each image is an RGB image of size 224$\times$224 depicting four objects. In
our dataset, an object can be described by four properties, namely shape
(cube, sphere, cylinder), material (metal or rubber), size (small or large),
and color (gray, red, blue, green, brown, purple, cyan, yellow). Data is
generated with respect to the ground truth rule and confounders as described
in the main paper. Relationships between objects are not used in ConCon. We
show several samples in Figure 6.

As we describe in detail in our main paper, the positive instances in the
dataset fulfill the ground truth rule ($g$) and the negative instances do not
($\neg g$). In addition, it is possible that a confounding feature must apply
($c_{t}$) or must not apply ($\neg c_{t}$), depending on the task (denoted by
subscript $t$) and the dataset variant (disjoint or strict). The ground truth
$g$ is defined as sphere and small cube. The confounders, in the order of
tasks, are $c_{1}$: blue, $c_{2}$: metal, and $c_{3}$: large. We summarize the
rules that should hold for respective tasks in Tables 3 (disjoint
configuration) and 4 (strict configuration).

| Positive | Negative
---|---|---
Task 1 | $g\wedge c_{1}\wedge\neg c_{2}\wedge\neg c_{3}$ | $\neg g\wedge\neg c_{1}\wedge\neg c_{2}\wedge\neg c_{3}$
Task 2 | $g\wedge\neg c_{1}\wedge c_{2}\wedge\neg c_{3}$ | $\neg g\wedge\neg c_{1}\wedge\neg c_{2}\wedge\neg c_{3}$
Task 3 | $g\wedge\neg c_{1}\wedge\neg c_{2}\wedge c_{3}$ | $\neg g\wedge\neg c_{1}\wedge\neg c_{2}\wedge\neg c_{3}$
Table 3: Rules disjoint: a confounding feature may only appear in the positive images of its respective task. A common rule is thus applied for all negative images across all tasks. | Positive | Negative
---|---|---
Task 1 | $g\wedge c_{1}$ | $\neg g\wedge\neg c_{1}$
Task 2 | $g\wedge c_{2}$ | $\neg g\wedge\neg c_{2}$
Task 3 | $g\wedge c_{3}$ | $\neg g\wedge\neg c_{3}$
Table 4: Rules strict: a confounding feature must appear in the positive
images but may not appear in the negative images of its respective task. The
confounding features may but are not required to appear in images of other
tasks. Figure 6: Samples for the ConCon dataset. We show the first four
samples for each dataset variant and task.

###  A.2 Code Framework

It is possible to generate further instances or make changes to the data
generation process by using our code repository. On the one hand, the logic in
the form of ground truth and confounders, which determine the data generation,
can be set in separate files. Note that the intended behavior for our data
generation consists of using simple confounders that are only required to be
satisfied on a single object. Here, our code could easily be adapted, for
example, to support the generation of images based on multi-object confounders
by changing the selection of object properties in the respective function. In
similar spirit, both CLEVR and ConCon are generated with the Blender graphics
simulation software, allowing for inclusion of entirely new attributes and
confounding features themselves. As such, extensions to e.g. novel color
choices are realizable at either trivial or modest amount of
specification/change in code, or respectively other already available
properties (such as relationships) could easily be leveraged in future dataset
creation and empirical investigation. Whereas these even more challenging
setups may certainly be of interest in the long term, we note that our
presently chosen configuration already highlights catastrophic behavior of
investigated methods and exposes insidiuous continual confounding - making it
desirable to solve our generated variant of ConCon first.

##  Appendix B Training Hyperparameters

In this section, we specify the training hyperparameters used for ConCon
training of NN and NeSy models.

#### NN training:

We use a ResNet-18 He et al. [2016], a convolution neural network based
architecture with skip connections. Following standard deep learning practice,
the model is trained with an Adam optimizer Kingma and Ba [2014] with a mini-
batch size of 64 and a learning rate of 0.001. The model was trained for 50
epochs.

#### NeSy training:

We use a neuro-symbolic concept learner (NeSy) Stammer et al. [2021] that is
equipped with a Slot attention model and a Set Transformer. The Slot-Attention
module of NeSy is pre-trained on the original CLEVR dataset Locatello et al.
[2020] for the purpose of extracting accurate neuro-symbolic intermediate
representations (e.g. disentangled slots for color and shape). We use the
(perfectly accurate as a consequence of simulation) structured information
from the embeddings of the attention module to then learn only the Set
Transformer for the ConCon dataset. The latter is trained using an Adam
optimizer Kingma and Ba [2014] with a mini-batch size of 64 and a learning
rate of 0.001. The model was trained for 150 epochs. We apply transformations
to resize the images to 128$\times$128 to fit the constraints set by prior
works Stammer et al. [2021].

We did not excessively tune hyperparameters as all model achieved good
solutions in principle - when they were supposed to in their baseline and such
a solution is achievable. As explained in the main body and discussed in much
more detail in the next section, observed failure cases are a systematic and
structural in nature - a consequence of continual learning set-up rather than
hyper-parameter values.

##  Appendix C Additional Empirical Corroboration

We now go into some further detail on our evaluation described in the main
paper. First, we comment on standard deviations across experimental seeds,
which have previously been omitted due to their negligible nature. We then
showcase a set of training curves to corroborate the consistency of our main
body’s results. Finally, additional empirical details for our main evaluation
are shown. Here, we further explain the models’ behaviors by considering
accuracies on the decomposed sets of positive and negative images, in addition
to their average.

###  C.1 Standard Deviations Across Experimental Repetitions with Random Seeds

In the main paper, we observed methods in the continual setting performing
well on the tasks they were trained on but completely failing on the
unconfounded dataset in the disjoint case and also not reaching the joint
accuracy in the strict case. To facilitate visual interpretation and focus on
the main insights, we had omitted reporting of standard deviations arising
from experimental repetition with (five) random seeds in the main body. We now
provide more details on why we consider these deviations to be negligible
overall, but in particular with respect to the drawn conclusions.

First, standard deviations are smaller than 1 percentage point of accuacy on
all diagonals (training and testing on the same task) except for the naive
training on the NN, where one outlier causes a higher deviation, and for the
cumulative setting, where it goes up to 3.65. For the latter, the higher
deviation is probably due to the model being trained on more than the task 3
data at the same time, therefore not being able to prioritize it as much as
other tasks, which almost reach perfect accuracy and thus have smaller
deviations. For both ER and the cumulative setting, all standard deviations
are smaller than 3. We again observe stronger deviations for the naive setup
as well as EWC. This fits with our observations of both approaches not
successfully combating forgetting, resulting in partly random behavior on
previous tasks.

Most importantly, standard deviations on the unconfounded datasets are fairly
small throughout. With the exception of the cumulative setting, we encounter
the largest standard deviation for the NN with EWC at around 2.18, with all
others being smaller. This highlights the consistency of the failure to learn
the ground truth rule. On the cumulative training setup, standard deviation on
the third task moves up to 5.78 for NN (less than 2 for NeSy). Whereas this is
in fact a large and relevant deviation from the mean, it is still a big
difference from the more than 20 points difference that is observed by
comparing the joint and the cumulative settings. This further emphasizes the
NN’s preference of learning confounders over the ground truth in continual
settings.

###  C.2 Training Converges on Confounded Data with Worse Results on
Unconfounded Data

Figure 7: Learning curves for the cumulative training setup. Recall that in
the cumulative setting the dataset being trained on grows over time, i.e. the
entire datasets from previous tasks are concatenated with the currently
observed one. This figure compares these curves to the test accuracy on the
unconfounded dataset (denoted by Unconf.), task-specific confounded datasets
(denoted by Conf.) and average accuracy calculated over accumulated confounded
tasks (denoted by Conf. Avg.) at the end of the training. Average accuracy
after $T$ tasks is given as: $A_{T}=\frac{1}{T}\sum_{t}^{T}a_{t}$. Observe
that in the strict case, performance on the unconfounded data does increase
during training but the final performance is still poor. We expect that this
can be explained by the further observation of the divergence between train
and validation curves in the strict case.

Recall that, in the strict setting, we saw the models and the NN in particular
achieve a high accuracy on the unconfounded dataset in the joint setting,
while training on the same data in a cumulative manner resulted in an accuracy
more than 20 points worse. This result is surprising since both setups are
trained on the exact same datasets. Hence, one might wonder whether the model
actually converged to a good accuracy during training in the cumulative setup.
As we can see in Figure 7, both models indeed converge at close to perfect
accuracy. For example, the NN ends at 100% train accuracy after task 1 and
99.5% accuracy after task 2. Training on the full accumulated data in task 3
peaks at 99.96% and ends at 99.68% accuracy. Despite this, performance on the
unconfounded dataset, while increasing with the number of tasks, remains
significantly worse than the train accuracy. In the main paper, we showed that
the easiest rule that can be learned to solve all confounded tasks jointly is
the ground truth rule. If that is true, why does the NN achieve a near perfect
train accuracy after training on all tasks but fail on the unconfounded data
where the ground truth also applies? By inspecting the validation and test
values of the confounded datasets, we find that, despite coming from the same
distribution of images, accuracies on these are considerably worse than for
the train images. It turns out that the cumulative setting has resulted in
both models overfitting far more severely to the train distribution. It thus
underperforms on both confounded data from the same distribution and
unconfounded data due to a general failure to fully generalize to new images.
We expect that more investigation of this overfitting will further illuminate
the problem of insidious continual confounding.

###  C.3 Models Mostly Fail on Positive Images in disjoint Settings

Figure 8: Accuracies for positive and negative images on disjoint. Test
accuracies on both the NN and NeSy for sequential training, EWC, ER, and
cumulative training. Rows on average accuracies are followed by results on
positive and negative images only. We can see that the prediction error is due
to the positive images which are incorrectly classified as negative. This is
consistent with our hypothesis of the model mostly basing its predictions on
the presence of the confounding features, none of which appear in any negative
images of the confounded tasks. For EWC and the cumulative setup, the high
performance on positive images after tasks 2 and 3 indicates that a
disjunction of confounding features is learned since this is satisfied in most
positive and only few negative images. (Best viewed in color) Figure 9:
Accuracies for positive and negative images on strict in resemblance to Figure
8. After task 1, performance on positive and negative images is relatively
balanced. Here, predictions based on the confounding feature perform well on
neither positive or negative images of other tasks since these feature may or
may not be present there. For tasks 2 and 3, the same reasoning applies but
since it is much more likely that this confounding feature is present, the
model tends to classify images as “positive”. This is due to the first
confounding feature (blue) having more alternative values (there are eight
colors) than the confounding features metal and large, which only have one
alternative value for their respective object property. (Best viewed in color)

Our main body’s evaluation on the disjoint dataset showed that performance on
the unconfounded dataset is not much better than random guessing, even when
forgetting is successfully avoided. In this part of the appendix, we go into
further detail on our evaluation to not only examine the average accuracies,
but also the models’ performance on the positive and negative images
individually. Through this analysis, we can then pinpoint the failure modes
even more precisely.

Figure 8 shows the decomposed results for both the NN and NeSy on all
individual data subsets. The first and firth rows display the average
accuracies of our models, repeating the same results also shown in our main
paper for convenience. The rows below these average figures highlight the
models’ performance on the positive and negative images only. Across all tasks
and methods, predictions on the negative set are mostly or even fully correct
and the classification errors are almost exclusively concentrated on the
positive set. In other words, the models seems to predict the negative class
too often, hence failing on the positive set. This can be explained by the
model predicting the positive class whenever the confounding feature is
present (it is never present on negative images of confounded tasks). As can
be seen from the results using ER, the model here manages to perform well on
all previous tasks as well, indicating that some combination of confounders is
learned.

We can get some more insights into the model behavior by relating to the
performance on the unconfounded dataset. For all methods, performance on both
positive and negative images is around 50% after the first task. Here, the
model appears to make predictions based on the first confounding feature,
blue, as in the unconfounded dataset this roughly matches the proportion of
images with and without a blue object. For the two following tasks, however,
almost all images are predicted as positive. Our results on ER, which show
good performance on all current and previously seen tasks, lead us to
hypothesize that the disjunction of all confounding features is learned for
both ER and in the (unlimited memory) cumulative setting. In any image of the
unconfounded dataset it is very likely that at least one confounding feature
is present, hence resulting in predictions for the positive class. For the
naive setup and EWC, the higher accuracies on the positive images can also
partly be explained by the data generation process: for both metal and large,
it is more likely for an unconfounded image to contain the respective features
than not to contain them.

With respect to NeSy, the same patterns can be observed. Performance on task 1
after training on task 2 is worse than for the NN, possibly indicating a
stronger focus on the confounder of the current task. We believe that the
marginally better accuracies on not yet seen tasks (forward transfer) can
partially be explained by the model learning some aspect of the ground truth
rule due to its neurosymbolic representation at the core.

Overall, results on the positive and negative images show that the negative
set of confounded tasks can be consistently predicted correctly. This is
entirely plausible and perhaps even expected as the negative set used in all
tasks follows shared rules for generation. In turn, errors are introduced by
incorrect predictions on positive images. The accuracies on the unconfounded
dataset conform with our hypothesis of the models learning a disjunction of
confounders, a rule which is mostly satisfied on positive unconfounded images.

###  C.4 Models Base Predictions on Task-Specific Confounders in strict
Settings

For the strict dataset, our evaluation in the main paper demonstrated the
problematic manifestation of insidious continual confounding, i.e. a model’s
performance when trained on all confounded task jointly is considerably higher
than when trained on the same data in a cumulative, continual setup. In the
latter, it largely fails.

Figure 9 shows the individually decomposed results for the strict dataset
variant, analogous to the analysis of the previous subsection. Again, the
model appears to have learned the first confounder after the first task.
However, due to this confounding feature being allowed to be present in other
tasks as well, this now leads to more balanced predictions on future tasks.
After training on tasks 2 and 3, most predictions fail on the negative instead
of the positive sets of images. This is again consistent with our hypothesis
of the model learning the task-specific confounder. To be precise, both the
confounders of task 2 and 3 describe a property, which, in our dataset, only
has two different possible values (metal or rubber; small or large). Since
these properties are determined randomly if not specified, there is only a
small probability of that feature not being present444Each of the four objects
in an unconfounded image has a chance of containing the material metal. The
probability of such an image not containing any metal object is therefore
quite low. The same idea applies analogously to large (on positive images, the
small sphere part of the ground truth rule also affects the distribution of
sizes).. Therefore, applying the rule based on the task-specific confounding
feature mostly classifies any such image as positive, no matter the class,
thus adding up to around 50% on average.

NeSy can be observed to perform less bad on negative images. It is possible
that it picks up on some parts of the ground truth rule more successfully than
the NN. Nevertheless, as we have seen in the main paper, the overall
performance on the unconfounded dataset still has a large margin of error.

In summary, the models mostly focus on the task-specific confounders in the
strict case. Whether such a model performs better on one class or the other is
then determined by the distribution of test images. In this strict variant of
ConCon, confounding features are likely to be present on any image of other
tasks and the unconfounded dataset, thus resulting in better predictions on
positive and worse prediction on negative images.

##  Appendix D ConCon Datasheet

Motivation

For what purpose was the dataset created? (e.g.was there a specific task in
mind? Was there a specific gap that needed to be filled? Please provide a
description.)

The ConCon dataset was created to evaluate confounding in a continual learning
setting. It consists of a learning task which can be solved by applying a
ground truth rule. There exists one unconfounded dataset where no confounder
is present and the ground truth determines the class assignment. Additionally,
there are two dataset variants, disjoint and strict, with three confounded
tasks each. Each task consists of its own dataset. In a continual learning
setting, a model is only trained on data from a single task at the same time,
going over all tasks in sequential order. For disjoint, the confounding
features are only present in their respective tasks. For strict, the
confounding features are only informative in their respective tasks, i.e. they
may only appear in one class of the confounded task but in any class of the
other tasks.

What (other) tasks could the dataset be used for? Are there obvious tasks for
which it should not be used?

The primary purpose of ConCon is the investigation of continual confounding.
It could also be used for other tasks, such as neuro-symbolic continual
learning.

Who created this dataset (e.g., which team, research group) and on behalf of
which entity ? (e.g., company, institution, organization)

The creators of the dataset are: Florian Peter Busch1,2, Roshni Kamath1,2,
Rupert Mitchell1,2, Wolfgang Stammer1,2, Kristian Kersting1,2,3,4 and Martin
Mundt1,2. The associated affiliations are: 1Department of Computer Science, TU
Darmstadt, Darmstadt, Germany, 2Hessian Center for AI (hessian.AI), Darmstadt,
Germany, 3German Research Center for Artificial Intelligence (DFKI),
Darmstadt, Germany and 4Centre for Cognitive Science, TU Darmstadt, Darmstadt,
Germany.

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

No funding was received for the creation of the dataset.

Any other comments?

No further comments.

Composition

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

Each instance is an image representing four geometric objects with certain
properties. An object can have any of three shapes (cube, sphere, cylinder),
be either small or large, have either metal or rubber material and be one of
eight colors (gray, red, blue, green, brown, purple, cyan, yellow).

How many instances are there in total (of each type, if appropriate)?

Each task consists of 9,000 instances. There is one unconfounded dataset and
both the disjoint and strict variants consist of three tasks each - resulting
in a total of 63,000 instances.

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

The dataset was generated synthetically. The entire space of possible samples
is given by any combination of properties of the four objects included in an
image. Additionally, images which do not conform to the dataset setup as
determined by the ground truth rule and confounding features are not sampled.
The dataset was sampled from the entire space of such permissible instances.
The positioning of objects in an image is also determined randomly, avoiding
images where an object is not visible (e.g. an object is hidden behind
another).

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? In either case, please provide a description.

Each instance consists of an image showing the objects as described
previously.

Is there a label or target associated with each instance? If so, please
provide a description.

Each instance fulfills or does not fulfill a (ground truth) rule. Thus,
instances could be seen as belonging to a positive or negative class.

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

No information is missing that is relevant to the intents and purposes of the
dataset’s creation. We follow the typical omission of nuanced graphics engine
simulation parameters.

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

Apart from sharing the rule determining the associated label, the dataset is
constructed to be confounded. In disjoint, there is a confounding feature that
is only present in the positive images of one specific tasks and does not
appear anywhere else in disjoint. In strict, these confounding features are
also present in the respective positive images and not in the same task’s
negative images but they may appear randomly in other tasks. The unconfounded
dataset is not confounded. No further relationships are made explicit.

Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.

Yes, each task consists of a training, validation and testing data split, with
3000, 750, and 750 instances, respectively. Additionally, we also have a
universal unconfounded dataset with the same split as the tasks. There is no
difference in the data generation of these splits.

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

There are no errors and no noise other that which is is implicitly caused by
the simulation process. There is a limited set of configurations resulting in
an instance. Nevertheless, this set is large but redundancy may occur. Note,
that this is on purpose to assure that models are capable of learning a fully
accurate solution as a baseline.

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.

The dataset is self-contained, however it does build on prior work. It is
built on the CLEVR framework (Johnson et al. 2017, CVPR), which has an open
source license that allows us to make these modifications. Data generation
also relies on the publicly accessible and free Blender software.

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals non-public
communications)? If so, please provide a description.

No.

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.

No.

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

No, the dataset only contains synthetic objects.

Does the dataset identify any subpopulations (e.g., by age, gender)? If so,
please describe how these subpopulations are identified and provide a
description of their respective distributions within the dataset.

No.

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? If so, please describe how.

No.

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If so,
please provide a description.

No.

Any other comments?

No further comments.

Collection Process

How was the data associated with each instance acquired? Was the data directly
observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey
responses), or indirectly inferred/derived from other data (e.g., part-of-
speech tags, model-based guesses for age or language)? If data was reported by
subjects or indirectly inferred/derived from other data, was the data
validated/verified? If so, please describe how.

This dataset was generated synthetically and by simulation according to
certain rules as explained previously. Data was generated using Blender and
not sampled from a real dataset.

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
How were these mechanisms or procedures validated?

The source code used to generate the datasets is available at:
https://github.com/ml-research/concon.

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?

Sampling was conducted in a probabilistic manner where unassigned object
properties are selected with a uniform probability. To ensure all objects are
recognizable, images where only a low amount of an object’s pixels are visible
(for example due to occlusion) are rejected and replaced with a new sample.
This generation procedure may thus result in non-uniform distributions in size
and shape attributes.

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?

Code was written and data generation conducted by the authors of this dataset
and its respective paper. No subcontractors or crowdworkers were involved as
all data is of simulated nature.

Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

The conception of the dataset spanned the course of roughly two months. The
data generation itself is resourceful and is on the scale of several hours on
consumer grade hardware.

Were any ethical review processes conducted (e.g., by an institutional review
board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

No, the entire dataset is purely synthetic and only consists of geometric
objects, so no ethical review was required.

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

No.

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?

No.

Were the individuals in question notified about the data collection? If so,
please describe (or show with screenshots or other information) how notice was
provided, and provide a link or other access point to, or otherwise reproduce,
the exact language of the notification itself.

No.

Did the individuals in question consent to the collection and use of their
data? If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.

No.

If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? If so,
please provide a description, as well as a link or other access point to the
mechanism (if appropriate).

No.

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted? If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.

No.

Any other comments?

No further comments.

Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? If so, please provide a
description. If not, you may skip the remainder of the questions in this
section.

No preprocessing was involved in the creation of this dataset.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data.

Since no preprocessing was conduced, the “raw” data and its label are the
primary content of this dataset.

Is the software used to preprocess/clean/label the instances available? If so,
please provide a link or other access point.

No preprocessing was involved in the creation of this dataset.

Any other comments?

No further comments.

Uses

Has the dataset been used for any tasks already? If so, please provide a
description.

Yes, we conducted an evaluation on continual confounding. Here, we showed that
machine learning models get confounded on the disjoint data. On the strict
data, we found out that performance on the unconfounded data is better when
training on data from all confounded tasks jointly when compared to the
continual learning setup.

Is there a repository that links to any or all papers or systems that use the
dataset? If so, please provide a link or other access point.

There is no such repository at the time of the creation of this dataset sheet.

What (other) tasks could the dataset be used for?

Please refer to the answer of the same question asked previously.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?

None that the authors are aware of, again referring to the synthetic nature of
the dataset consisting of simulated simple objects (like colored cubes and
squares) coupled with logical rules.

Are there tasks for which the dataset should not be used? If so, please
provide a description.

Please refer to the previously asked similar question.

Any other comments?

No further comments.

Distribution

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created? If so, please provide a description.

Yes, the dataset is publicly available at Zenodo and the source code is
available at Github.

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub) Does the dataset have a digital object identifier (DOI)?

We use the third-party open-access repository Zenodo555https://zenodo.org/.
Zenodo features unique digital object identifiers. The Zenodo DOI for our
dataset is: 10.5281/zenodo.10630482.

When will the dataset be distributed?

The dataset is made available on 9th February, 2024.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

The dataset will be distributed under the BSD 2-Clause “Simplified” License.

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

No.

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

No.

Any other comments?

No further comments.

Maintenance

Who will be supporting/hosting/maintaining the dataset?

We use Zenodo to host the dataset, a reputable open-access repository where
the dataset has a high likelihood to persist long into the future. Maintaining
is carried out by the authors and affiliated groups.

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

The creators of the datasets: Florian Peter Busch and Roshni Kamath can be
contacted at<EMAIL_ADDRESS>and roshni.kamath@tu-
darmstadt.de respectively.

Is there an erratum? If so, please provide a link or other access point.

There presently is no erratum. However, Zenodo offers an erratum functionality
should it become relevant in the future.

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?

Currently, no immediate updates are envisioned.

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were individuals in question
told that their data would be retained for a fixed period of time and then
deleted)? If so, please describe these limits and explain how they will be
enforced.

The dataset does not relate to people.

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to users.

Yes, Zenodo supports access of old versions using unique Digital Object
Identifiers (DOI).

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
other users? If so, please provide a description.

The code for data generation is publicly available on Github at
https://github.com/ml-research/concon. Validation/Verification of future
contributions will not be in the scope of the authors.

Any other comments?

No further comments.