# Treatment Learning Causal Transformer for Noisy Image Classification

Chao-Han Huck Yang∗
Georgia Institute of Technology
<EMAIL_ADDRESS>&I-Te Danny Hung∗
Columbia University
<EMAIL_ADDRESS>&Yi-Chieh Liu∗
Georgia Institute of Technology
<EMAIL_ADDRESS>
&Pin-Yu Chen
IBM Research
<EMAIL_ADDRESS>
###### Abstract

Current top-notch deep learning (DL) based vision models are primarily based
on exploring and exploiting the inherent correlations between training data
samples and their associated labels. However, a known practical challenge is
their degraded performance against “noisy” data, induced by different
circumstances such as spurious correlations, irrelevant contexts, domain
shift, and adversarial attacks. In this work, we incorporate this binary
information of “existence of noise” as treatment into image classification
tasks to improve prediction accuracy by jointly estimating their treatment
effects. Motivated from causal variational inference, we propose a
transformer-based architecture, Treatment Learning Causal Transformer (TLT),
that uses a latent generative model to estimate robust feature representations
from current observational input for noise image classification. Depending on
the estimated noise level (modeled as a binary treatment factor), TLT assigns
the corresponding inference network trained by the designed causal loss for
prediction. We also create new noisy image datasets incorporating a wide range
of noise factors (e.g., object masking, style transfer, and adversarial
perturbation) for performance benchmarking. The superior performance of TLT in
noisy image classification is further validated by several refutation
evaluation metrics. As a by-product, TLT also improves visual salience methods
for perceiving noisy images.

##  1 Introduction

Although deep neural networks (DNNs) have surpassed human-level “accuracy” in
many image recognition tasks (Ronneberger et al., 2015; He et al., 2016; Huang
et al., 2017), current DNNs still implicitly rely on the _assumption_ (Pearl,
2019) on the existence of a strong correlation between training and testing
data. Moreover, increasing evidence and concerns (Alcorn et al., 2019) show
that using the correlation association for prediction can be problematic
against noisy images (Xiao et al., 2015), such as pose-shifting of identical
objects (Alcorn et al., 2019) or imperceptible perturbation (Goodfellow et
al., 2015). In practice, real-world image classification often involves rich,
noisy, and even chaotic contexts, intensifying the demand for generalization
in the wild. Putting in a unified descriptive framework,

Figure 1:  _(a)_ An example of deployed causal graphical model (CGM), where
$\mathbf{Z}$ denotes unobservable confounder variable (e.g., the concept of
“cat”), $\mathbf{X}$ denotes a noisy observation of confounder (e.g., an image
can still be recognized as a cat), $\mathbf{y}$ denotes outcome (e.g., a
label), and $\mathbf{t}$ denotes the information of a binary treatment (e.g.,
the existence of extra semantic patterns or additive noise; thus, it is equal
to $0$ or $1$), which is observable during training and unobservable during
testing time. _(b)_ Images with “cat” labels, where (i) and (ii) share the
same context of “indoor”; (iii) shows a noisy setup of (ii) undergoing
additive Gaussian perturbation; (iv) shows another setup of introducing extra
noisy semantic patterns (e.g., “waterside”) in NICO (He et al., 2020) noisy
images dataset.

To address machine perception against noisy images, we are inspired by how
human performs visual recognition. Human’s learning processes are often mixed
with logic inference (e.g., a symbolic definition from books) and
representation learning (e.g., an experience of viewing a visual pattern). One
prominent difference between current DNNs and human recognition systems is the
capability in causal inference. Mathematically, causal learning (Pearl, 1995a;
Peters et al., 2014) is a statistical inference model that infers beliefs or
probabilities under uncertain conditions, which aims to identify latent
variables (called “confounders”) that influence both intervention and outcome.
The unobserved confounders may be abstract in a cognitive-level (e.g.,
concepts) but could be observed via their noisy view in the real-world (e.g.,
objects). For instance, as shown in Fig. 1 _(a)_ , confounder learning aims to
model a prediction process by finding a representation (e.g., “cat”) and
avoiding relying on irrelevant patterns (e.g., “waterside”). Intuitively, with
causal modeling and confounder inference, correct prediction can be made on
noisy inputs, where the generative estimation process, such as causal effect
variational autoencoder (CEVAE) (Louizos et al., 2017), affects multiple
covariates for predicting data profiles. In this work, we aim to incorporate
the effects of causal confounder learning to image classification, as
motivated by cognitive psychology for causal learning. Specifically, we use
the attention mechanism for noise-resilience inference from patterns. We
design a novel sequence-to-sequence learning model, Treatment Learning Causal
Transformer (TLT), which leverages upon the conditional query-based attention
and the inference power from a variational causal inference model.

Our TLT tackles noisy image classification by jointly learning to a generative
model of $\mathbf{Z}$ and estimating the effects from the treatment
information ($\mathbf{t}$), as illustrated in Fig. 1 (a). This model consists
of unobservable confounder variables $\mathbf{Z}$ corresponding to the ground-
truth but inaccessible information (e.g., the ontological concept (Trampusch
and Palier, 2016) of a label), input data $\mathbf{X}$ from a noisy view of
$\mathbf{Z}$ (e.g., images), a treatment (Pearl et al., 2016) information
$\mathbf{t}$ given $\mathbf{X}$ and $\mathbf{Z}$ (e.g., secondary information
as visual patterns and additive noise without directly affecting our
understanding the concept of “cat”), and a classification label $\mathbf{y}$
from the unobservable confounder. Built upon this causal graphical model, our
contributions are:

  * •

A transformer architecture (TLT) for noisy image classification are presented,
which is based on a treatment estimation architecture and a causal variational
generative model with competitive classification performance against noisy
image.

  * •

We further curated a new noisy images datasets, Causal Pairs (CPS), to study
generalization under different artificial noise settings for general and
medical images.

  * •

We use formal statistical refutations tests to validate the causal effect of
TLT, and show that TLT can improve visual saliency methods on noisy images.

##  2 Related Work

Noisy Image Classification. Prior works on noisy images classification have
highlighted the importance of using generative models (Oquab et al., 2014) to
ameliorate the negative learning effects from noisy data. Xiao _et al._ (Xiao
et al., 2015) leverage a conditional generative model (Sohn et al., 2015) to
capture the relations among images and noise types from online shopping
systems. Direct learning from noisy data is another approach by using
statistical sampling (Han et al., 2019; Li et al., 2017) and active learning
(Gal et al., 2017) for performance enhancement. Meanwhile, new noisy images
dataset and evaluation metrics (He et al., 2020) on context independence have
been proposed, such as Strike simulator (Alcorn et al., 2019) for synthesizing
pose-shifting images and NICO (He et al., 2020; Zhang et al., 2021; Liu et
al., 2021) as the open-access noisy image dataset. NICO further aims to
highlight the importance of incorporating a statistical inference (e.g.,
causal model) for improved image classification with large-scale noisy
context-patterns (e.g., an image shows “cat in waterside” but given a single
label of “cat”). However, different from context-wise noise in NICO, modeling
sizeable artificial noise in images is crucial yet remains unexplored. In this
work, we create a new image dataset containing various artificial noise and
use the NICO (He et al., 2020) with a generative causal model for performance
benchmarking.
Causal Learning for Computer Vision. Many efforts (Pickup et al., 2014; Fire
and Zhu, 2016; Lebeda et al., 2015; Fire and Zhu, 2013) have leveraged upon
causal learning to better understand and interpret toward vision recognition
tasks. Lopez-Paz _et al._ (Lopez-Paz et al., 2017) propose utilizing DNNs to
discover the causation between image class labels for addressing the
importance of this direct causal relationship affecting model performance and
context grounding. Incorporating causal analysis and regularization showed
improved performance in generative adversarial models such as Causal-GANs
(Kocaoglu et al., 2017; Bahadori et al., 2017). However, infusing causal
modeling and inference to DNN-based image recognition systems is still an open
challenge. For instance, in previous works (Lopez-Paz et al., 2017; Yang et
al., 2019), researchers focus on modeling a direct causal model (DCM) (Pearl
et al., 2016) for visual learning. The DCMs treat a visual pattern (e.g.,
texture) as a cause visual representation (e.g., patterns of the “cat”) and
barely incorporate additional label information (e.g., context) or apply noise
as a treatment in causal analysis. In recent works, causal modeling also show
promising results in a large-scale computer vision task, such scene graph
(Tang et al., 2020) generation, visual and language learning (Qi et al., 2020;
Agarwal et al., 2020; Abbasnejad et al., 2020), and semantic segmentation
(Zhang et al., 2020). The work of Chalupkaet al. (Chalupka et al., 2014) is
closer to our work by deploying interventional experiments to target causal
relationships in the labeling process. However, modeling the aforementioned
treatment effects and designing efficient learning models are still not fully
explored (Pearl, 2019).
Causal Inference by Autoencoder. Recently, classical causal inference tasks,
such as regression modeling (Bühlmann et al., 2014), risk estimation (Pearl,
2019), and causal discovery (Monti et al., 2020), have been incorporated with
deep generative models (Rezende et al., 2014) and attained state-of-the-art
performance (Shalit et al., 2017; Louizos et al., 2017). These generative
models often use an encoder-decoder architecture to improve both logic
inference and features extracted from a large-scale dataset with noisy
observations. TARNet (Shalit et al., 2017) is one foundational DNN model
incorporating causal inference loss from a causal graphical model (CGM) and
feature reconstruction loss jointly for linear regression, showing better
results compared with variational inference models (Kingma and Welling, 2013).
Inspired by the CGM of TARNet (Shalit et al., 2017), causal-effect variational
autoencoder (CEVAE) was proposed in (Louizos et al., 2017; Yang et al., 2021)
for regression tasks, which draws a connection between causal inference with
proxy variables and latent space learning for approximating the hidden and
unobservable confounder by the potential outcome model from Rubin’s causal
inference framework (Imbens and Rubin, 2010; Rubin, 1974). Our proposed causal
model in TLT shares a similar CGM with CEVAE but has a different training
objective, probabilistic encoding, and specific design for visual recognition,
such as the use of attention mechanism.

##  3 TLT: Treatment Learning Causal Transformer

Table 1: _Causal hierarchy_ (Pearl, 2009): questions at level $i$ can only be
answered if information from the same or higher level is available.

Level Activity PGM Example ($\mathbb{I}$) Association Observing $P(y|x)$
ResNet (He et al., 2016) ($\mathbb{II}$) Intervention Intervening
$P(y|do(x),z)$ TLT (ours)

###  3.1 Modeling under Causal Hierarchy Theorem

To model a general image classification problem with causal inference, we
introduce Pearl’s _causal hierarchy Theorem_ (Bareinboim et al., 2020;
Shpitser and Pearl, 2008; Pearl, 2009) as shown in Tab. 1, with a non-causal
classification model and a causal inference model. Non-causal model is in
level ($\mathbb{I}$) of causal hierarchy, which associates the $outcome$
(prediction) to the input directly by $P(y|x)$ from supervised model such as
ResNet (He et al., 2016). Non-causal model could be unsupervised by using
approximate inference such as variational encoder-decoder (Bahuleyan et al.,
2018) with two parameterized networks, $\Theta$ and $\Phi$. The association-
level (non-causal) setup in the causal hierarchy can solve visual learning
tasks at level ($\mathbb{I}$), such as non-noisy image classification.

For noisy image classification, we argue that the problem setup is elevated to
level ($\mathbb{II}$) of the causal hierarchy, requiring the capability of
confounder learning and the $do$-$calculus$ (Pearl, 2019) (refer to causal
inference foundations supplement A). We first make a formal definition on a
pair of $i^{th}$ query $(x_{i},y_{i})$ including a noisy image input ($x_{i}$)
and its associated label ($y_{i}$). Suppose for every noisy image, there
exists a clean but inaccessible image ($\tilde{x}_{i}$) and treatment
information ($t_{i}$), where the intervened observation is modeled as
$P(x_{i})=P(do(\tilde{x}_{i}))\equiv P(\tilde{x}_{i}|t_{i})$, and $t_{i}$
encodes full information of the intervention through the do-operator notation
$do(\cdot)$. The corresponding confounder $z_{i}$ follows
$P(z_{i})=P(\tilde{x}_{i},t_{i},\tilde{z}_{i})$, where $\tilde{z}_{i}$ is the
unobservable part (e.g., undiscovered species of “cat” but belong to its
ontological definition) of the confounder. To make a prediction ($y_{i}$) of a
noisy input of ($x_{i}$), we could have the intervened view of the question
by:

$P(y_{i}|x_{i})=P(y_{i}|do(\tilde{x}_{i}),z_{i})=P(y_{i}|\tilde{x}_{i},t_{i},z_{i})$
(1)

with do-operator in level ($\mathbb{II}$) of the causal hierarchy. Based on
the causal hierarchy, we could use the model with the proxy variables
($z_{i},t_{i}$) in the higher level ($\mathbb{II}$) to answer the question in
equal or lower level. Next, we introduce our training objective using an
encoder-decoder architecture to reparameterize the aforementioned proxy
variables for causal learning.

Figure 2: The encoder (inference network) structure of our proposed causal
transformer. We leverage bilinear fusion (BF) for $q(z|x,y,t)$ instead of
concatenation (Louizos et al., 2017), and decoding conditional queries
$H_{z}\sim q(y|x,t)$ and encoding features $H_{x}\sim p(x)$ as keys and values
to conduct attention. Decoder model is shown as Fig. 2 (a) with potential
outcome modeling (Imbens and Rubin, 2010; Rubin, 1974) from $p(z)$.

###  3.2 Training Objective of TLT

We build our TLT model based on the foundational framework of conditional
variational encoder-decoder (CVED) (Bahuleyan et al., 2018; Kingma and
Welling, 2013), which learns a variational latent representation $z_{i}$ from
data $x_{i}$ and conditional information (e.g., label $y_{i}$) for
reconstruction or recognition. To effectively learn visual causal pattern
recognition, our TLT model uses variational inference to approximate the
complex non-linear relationships involving: the pair probability
($p(x_{i},z_{i})$), the treatment likelihood $P(t_{i})$, the model outcome
$p(y_{i})$, and the joint distribution $p(z_{i},x_{i},t_{i},y_{i})$.
Specifically, we propose to characterize the causal graphical model in Fig. 1
_(a)_ as a latent variable model parameterized by a DNN encoder-decoder as
shown in Fig. 6 (in Appendix A). Note that TLT uses an advanced decoding
method $p(a_{i})=F_{T}(H_{x},H_{z}\sim P(x_{i}))$ for approximating $p(z_{i})$
from $p(x_{i})$ based on the attention ($F_{T}$) from transformer (Vaswani et
al., 2017), which will be detailed in Sec. 3.3.

First, we assume the observations factorize conditioned on the latent
variables and use an _general_ inference network (encoder) which follows a
factorization of the true posterior. For the model network (decoder), instead
of conditioning on observations, we approximate the latent variables $z$. For
vision tasks, $x_{i}$ corresponds to a noisy input image indexed by $i$,
$t_{i}\in\\{0,1\\}$ corresponds to the treatment assignment, $y_{i}$
corresponds to the outcome and $z_{i}$ corresponds to the latent hidden
confounder. Note that general formation of an approximation outcome ($\delta$)
is modeling by $\delta_{i}=t_{i}\cdot y_{i}+(1-t_{i})\cdot y_{i}$ as potential
outcome model (Imbens and Rubin, 2010; Holland, 1986) with its foundation over
the causal inference. Next, each of the corresponding factors is described as:

$\displaystyle p(z_{i})=\prod_{z\in
z_{i}}\mathcal{N}(z|0,1);~{}~{}p(x_{i}|z_{i})=\prod_{x\in
x_{i}}p(x|z_{i});~{}~{}p(t_{i}|z_{i})=Bern(\sigma(f_{1}(z_{i})));$
$\displaystyle
p(y_{i}|z_{i},t_{i})=\sigma(t_{i}f_{2}(z_{i})+(1-t_{i})f_{3}(z_{i}))$

with $\mathcal{N}(\mu,\sigma^{2})$ denoting a Gaussian distribution with mean
$\mu$ and variance $\sigma^{2}$, $p(x|z_{i})$ being an appropriate probability
distribution, $\sigma(.)$ being a logistic function, and $Bern(.)$ denotes the
probability of success of a Bernoulli random variable. Each of the $f_{k}(.)$
function is an Adaptive Average Pooling plus Linear layer parameterized by its
own parameters $\theta_{k}$ for $k=\\{1,2,3\\}$. Here $y_{i}$ is tailored for
categorical classification problems, but our formulation can be naturally
extended to different tasks. For example, one can simply remove the final
$\sigma(.)$ layer of $p(y_{i}|z_{i},t_{i})$ for regression tasks.

Our TLT inference network (encoder), as illustrated in Fig. 2, aims to learn
meaningful causal representations in the latent space. As we can see from Fig.
1 _(a)_ , the true posterior over $z\in\mathbf{Z}$ depends on
$x\in\mathbf{X}$, $t$, and $y$. We are required to know the treatment
assignment $t$ along with its outcome $y$ prior to inferring the distribution
over $z$. Therefore, unlike variational encoders, which simply passes the
feature map directly to latent space (the top path in our encoder), the
feature map extracted from a residual block is provided to the other switching
(the lower and middle paths in our encoder), which provides posterior
estimates of treatment $t_{i}$ and outcome $y_{i}$. The switching mechanism
(binary selection based on the treatment information of $t_{i}$ = 0 or 1) and
its alternative loss training have been widely used in TARNet (Shalit et al.,
2017) and CEVAE (Louizos et al., 2017) with theoretical and empirical
justification. We employ the distribution by the switching mechanism:

$\displaystyle q(t_{i}|x_{i})=Bern(\sigma(g_{1}(x_{i})));$ (2) $\displaystyle
q(y_{i}|x_{i},t_{i})=\sigma(t_{i}g_{2}(x_{i})+(1-t_{i})g_{3}(x_{i})),$ (3)

with each $g_{k}$ being a neural network approximating $q(t_{i}|x_{i})$ or
$q(y_{i}|x_{i},t_{i})$. They introduce auxiliary distributions that help us
predict $t_{i}$ and $y_{i}$ for new samples. To optimize these two
distributions, we add an auxiliary objective to our overall model training
objective over $N$ data samples:

$\displaystyle\mathcal{L}_{aux}=\sum^{N}_{i=1}(\log
q(t_{i}=t_{i}^{*}|x_{i}^{*})+\log q(y_{i}=y_{i}^{*}|x_{i}^{*},t_{i}^{*})),$
(4)

where $x_{i}^{*}$, $t_{i}^{*}$ and $y_{i}^{*}$ are the observed values in
training set. Since the true posterior over $z$ depends on $x$, $t$ and $y$,
finally we employ the posterior approximation below:

$\displaystyle
q(z_{i}|x_{i},y_{i},t_{i})=\prod_{z_{i}}\mathcal{N}(\bm{\mu}_{i},\bm{\sigma^{2}}_{i})$
(5)
$\displaystyle\bm{\mu}_{i}=t_{i}\bm{\mu}_{t=1,i}+(1-t_{i})\bm{\mu}_{t=0,i},~{}~{}\bm{\sigma^{2}}_{i}=t_{i}\bm{\sigma^{2}}_{t=1,i}+(1-t_{i})\bm{\sigma^{2}}_{t=0,i}$
$\displaystyle\bm{\mu}_{t=0,i}=g_{4}\circ g_{0}(x_{i},y_{i}),\hskip
5.69046pt\bm{\sigma^{2}}_{t=0,i}=\sigma(g_{5}\circ g_{0}(x_{i},y_{i}))$
$\displaystyle\bm{\mu}_{t=1,i}=g_{6}\circ g_{0}(x_{i},y_{i}),\hskip
5.69046pt\bm{\sigma^{2}}_{t=1,i}=\sigma(g_{7}\circ g_{0}(x_{i},y_{i}))$

where $g_{k}$ again denotes neural network approximation, and
$g_{0}(x_{i},y_{i})$ is a shared, bilinear-fusioned representation of $x$, $t$
and $y$. More specifically, we multiply the feature map with approximated
posterior $q(y_{i}|x_{i},t_{i})$ without logistic function $\sigma$ to get
$g_{0}(x_{i},y_{i})$. Finally, we can have the overall training objective for
the inference and model networks. The variational lower bound of TLT to be
optimized is given by:

$\displaystyle\mathcal{L}_{TLT}=\mathcal{L}_{aux}+\sum^{N}_{i=1}\mathbb{E}_{q(z_{i}|x_{i},t_{i},y_{i})}[\log
p(x_{i},t_{i}|z_{i})+\log p(y_{i}|t_{i},z_{i})+\log p(z_{i})-\log
q(z_{i}|x_{i},t_{i},y_{i})].$ (6)

As shown in Fig. 6 (in Appendix A), we could model $q(t|x)\doteq p(t)$ to
access the treatment information directly for training to guide one
corresponding sub-network in Fig. 2; for testing, $q(t|x)$ could be inferred
by a given input $x$ without knowing treatment information from an
unsupervised perspective.

###  3.3 Attention mechanism of TLT

Attention mechanism is one of the human learning components to capture global
dependencies for discovering logical and causal relationships (Nauta et al.,
2019) from visual patterns in the cognitive psychology community (Chen et al.,
2015). Transformer (Vaswani et al., 2017) based attention mechanism has,
recently, shown its connection from the sequential energy update rule to
Hopfield networks (Ramsauer et al., 2020), which stands for a major framework
to model human memory. With the intuition on leveraging human-inspired
attention upon inference from noisy images, we incorporate a new type of
Transformer module for the proposed causal modeling, which explicitly model
all pairwise interactions between elements in a sequence. The idea is to learn
the causal signal (Lopez-Paz et al., 2017) via self-attention setup, where we
set the interference signal ($H_{z}$) for learning query and image features
($H_{x}$) for learning key and value. As shown in Fig 2, we use a feature map
with a ResNet34 (He et al., 2016) encoder extracting from input image
$p(x_{i})$ feeding into keys ($K$) and value ($V$) with queries $q(y_{i})$
from Eq. (3):

$\displaystyle Q=$ $\displaystyle~{}\operatorname{unroll}\left(F_{Q}(H_{z}\sim
q(y_{i}|x_{i},t_{i}))\right);~{}~{}K=\operatorname{unroll}\left(F_{K}(H_{x}\sim
p(x_{i}))\right)$ (7) $\displaystyle V=$
$\displaystyle~{}\operatorname{unroll}\left(F_{V}(H_{x}\sim
p(x_{i}))\right);~{}~{}a_{i}=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(8)

where $F_{Q}$, $F_{K}$, $F_{V}$ are convolutional neural networks and $d_{k}$
is dimension of keys. Finally, we model $q(z_{i})$ by using $q(t_{i}|x_{i})$
and $p(a_{i}|x_{i})$ with the causal two model extended from Eq. (5) for
approximating posterior distribution $p(z_{i})$:

$\displaystyle p(z_{i})\leftarrow
q(z_{i}|x_{i},a_{i},y_{i},t_{i})=\prod_{z_{i}}\mathcal{N}(\bm{\mu}_{i},\bm{\sigma^{2}}_{i}).$
(9)

We also have conducted ablation studies on architecture selection and required
parameters with respect to supervised learning (He et al., 2016), attention
networks (Vaswani et al., 2017), and causal model (Shalit et al., 2017) in
supplement B to validate our model design of TLT. To sum up, the proposed
causal architecture attains the best performance with the same amount of
parameters.

##  4 Evaluating Causal Effects on Noisy Images

In this section, we introduce noisy image datasets and conduct statistical
refutation tests on TLT to evaluate its causal effect based on the CGM in Fig.
1 _(a)_. That is, we provide an affirmative answer to whether there exist
causal effects in the studied noisy image classification tasks.

###  4.1 Estimate Causal Effects

Estimation of expected causal effects is one general approach (Pearl, 2019,
2009; Louizos et al., 2017) to evaluate whether a CGM (from a logic
hypothesis) is valid on the selected test dataset. The underlying graphical
model will undergo a series of randomization tests of graphical connection and
sub-set sampling to measure its estimation errors on estimating causal
effects. In general, a causal model is reliable with the CGM when exhibiting a
lower absolute error on the causal effects. In this work, we use average
treatment effects (ATE), as used in prior arts (Louizos et al., 2017), for
comprehensive analysis.
Average Treatment Effects (ATEs). In the binary treatment setting (Pearl,
1995a), for the $i$-th individual and its associated model outcome $y_{i}$
considering the treatment effect, the ATE is calculated by:

$\displaystyle y_{i}$
$\displaystyle=y_{t_{i}=0,i}\left(1-t_{i}\right)+y_{t_{i}=1,i}(t_{i}),$ (10)
$\displaystyle ATE$
$\displaystyle=|\mathbb{E}\left[y_{i}=y^{*}_{i}|t^{*}_{i}=1\right]-\mathbb{E}\left[y_{i}=y^{*}_{i}|t^{*}_{i}=0\right]|,$
(11)

where $y_{t_{i},i}$ denotes the prediction with estimated treatment
$t_{i}\in\\{0,1\\}$. $y^{*}_{i}$ and $t^{*}_{i}$ are the observations. The ATE
is taken over all subjects. From (Greenland et al., 1999), these metrics
cannot be properly estimated if there are confounding variables in the system.
On the other hand, Pearl (Pearl, 1995a) introduces the “do-operator” (Pearl,
2019) on treatment to study this problem under intervention. The $do$ symbol
removes the treatment $t$ from the given mechanism and sets it to a specific
value by some external intervention. The notation $P(y|do(t))$ denotes the
probability of $y$ with possible interventions on treatment. Following Pearl’s
back-door adjustment formula (Pearl, 2009) and the CGM in Fig. 1, it is proved
in (Louizos et al., 2017) that the causal effect for a given binary treatment
$t$, a proxy variable $x$, an outcome $y$ and a confounding variable $z$ can
be evaluated by (similarly for $t=0$):

$\begin{array}[]{l}{p(y|x,do(t=1))}={\int_{z}p(y|x,t=1,z)p(z|x)dz}\end{array}$
(12)

To intervene the information of $t$ ($do(t)$), flipping errors (Louizos et
al., 2017) with different rates (see supplement C) are applied to change the
$t_{i}$ label(s) (Paszke et al., 2017) in our experiments in Section 4. The
proposed CGM and its associated TLT show resilient ATE estimation under
statistical refutations.

Visual Patterns in the Intervention Level ($\mathbb{II}$). We clarify two
common scenarios, noisy context and under perturbation, in the intervention
level ($\mathbb{II}$) for noisy image classification. As shown in Tab. 2, the
treatment information ($t$) is binary with an accessible noisy input $x$ and
inaccessible ontological (clean) representation $\tilde{x}$ from Eq. (1) for
visual pattern modeling. Next, we introduce datasets in the regime of the case
1 and 2 for our experiments in this work.

Table 2: Applying causal modeling on noisy images classification. Case 1: perfect labeled visual pattern (Hlab) with secondary patterns (Hiid); Case 2: original labeled images (Hori) under an additive perturbation (Fper).  Treatment | $\tilde{x}$ | $do(\tilde{x})$ |  $t$=1 or 0
---|---|---|---
1. Context | Hlab | Hlab\+ Hiid | Additional patterns (e.g., “waterside”) (1) or not (0)
2. Perturbation | Hori | Fper(Hori) | Artificial noise (e.g., Gaussian) (1) or not (0)

###  4.2 Case 1: NICO Dataset with Noisy Extra Visual Patterns

NICO (He et al., 2020) is a large-scale and open-access benchmark dataset for
noisy image classification, which is motivated by studying non-independent
image classification with causal modeling. The NICO dataset labels images with
both main concepts (e.g., “cat”) and contexts as sub-labels (e.g., “water”).
NICO is constructed by two super-classes: “animal” and “vehicle”, with 10
classes for “animal” and 9 classes for “vehicle”. In total, NICO contains 19
classes, 188 contexts, and 25,000 images. The design intuition of NICO is to
provide a causal modeling benchmark for large-scale image classification. The
authors evaluate several major image classification dataset (e.g., ImageNet,
Pascal, and MS-COCO) and found out the auxiliary context information
(treatment) is much random and inaccurate from statistical measurement for
structuring validated causal inference. By selecting different contexts of the
concept, testing data distribution can be unknown and different from training
data distribution, which can be used to evaluate a causal inference model.

In our experiments, we follow the standard NICO evaluation process (He et al.,
2020), where a concept is incorporated with two contexts. We further use
context as treatment in the intervention level as in Case 1 of Tab. 2. One
context is the attribute of concept ($t=1$) while another context is the
background or scene of a concept ($t=0$).

###  4.3 Case 2: Curated Causal Pairs (CPS) Dataset with Additive Artificial
Noises

Despite many efforts in providing benchmark datasets for causal inference on
non-vision tasks (Hill, 2011; LaLonde, 1986; Hoyer et al., 2009; Peters et
al., 2014), visual causal data collection is relatively limited to bare causal
effect evaluation with conditional visual treatments (Lopez-Paz et al., 2017).
Motivated by the perturbation-based causation studies testing biological
network and the efforts from NICO, we further curate two datasets from public
sources, named causal pairs (CPS), by using a diverse set of image
perturbation types as treatment (i.e., Case 2 in Tab. 2).

We select two representative datasets, Microsoft COCO (Lin et al., 2014), and
a medicine dataset, Decathlon (Simpson et al., 2019), to create our CPS
datasets. Each CPS contains pairs of original and perturbed images, as well as
five different perturbation types described in Sec. 4.4. Table. 3 summarizes
the NICO and our CPS datasets. Next, we introduce how to generate noisy images
in CPS.

Figure 3: Illustration of our generated CPS dataset for noisy image
classification. We randomly selected $\mathbf{50\%}$ of labeled images from
both datasets and applied visual modifications to generate interventional
observations. We selected similar object classes by 1,000 human surveys. Left:
three causal pairs – giraffe/elephant, fire-hydrant/stop-sign, and
motorcycle/bike. From left to right in CPS, the visual treatments are: (a)
original input image, (b) image scrambling, (c) neural style transfer; (d)
adversarial example. We further discuss masking intervention effects used in
(Lopez-Paz et al., 2017; Yang et al., 2019) on general subjects by (e) object
masking; and (f) background refilling. Right: a demonstration of Lung Tumours
in Decathlon of the same format. Table 3: Comparison of noisy image
classification datasets: CPS with perturbation as a treatment (see Sec. 4.4)
and NICO with noisy context (e.g., “indoor”).

Dataset Treatment (Binary Information) Numbers Super-classes Total classes CPS
(ours) Receiving artificial noise (or not) 13,752 General / Medical 16 NICO
Existing context-wise pattern (or not) 25,000 Animal / Vehicle 19

Super-class 1: Generating Noisy General Objects. To generate CPS dataset from
MS-COCO (Lin et al., 2014) for general super-class, we selected six similar
object classes that could possibly result in confusing interpretation and
recognition by human psychology studies (Reed, 2012; Musen and Treisman, 1990)
(e.g., giraffe and elephant, etc.). We conduct a survey with 1,000 volunteers
from Amazon mechanical turk (Turk, 2012) and pick the top-3 similarity label
pairs. Specifically, we format three different common causal pairs, namely
giraffe-elephant (CPS1) with 3316 images, stop sign-fire hydrant (CPS2) with
2419 images, and bike-motorcycle (CPS3) with 4729 images, where the dataset is
visualized in Fig. 3 (a).
Super-class 2: Generating Noisy Medical Images. For the medical super-class,
we use an identical setting with 2630 training and 658 test CT images for ten
different types (total classes) of human disease from Decathlon (Simpson et
al., 2019), which includes: (1) Liver Tumours; (2) Brain Tumours; (3)
Hippocampus; (4) Lung Tumours; (4) Prostate; (5) Cardiac; (6) Pancreas Tumour;
(7) Colon Cancer; (8) Hepatic Vessels, and (10) Spleen. More details and
visualization (Fig. 3 (b)) about this dataset are given in supplement B. From
these two super-classes, we randomly selected $\mathbf{50\%}$ of these labeled
images and applied visual modifications to generate interventional
observations. Each generated image is assigned with a binary treatment
indicator vector $t_{i}$, where its $i$-th element denotes the binary
treatment label according to the $i$-th visual modification.

###  4.4 Visual Perturbation (Treatment) in CPS

We employ five distinct types of image modification methods as independent
intervention variables: (i) image scrambling; (ii) neural style transfer;
(iii) adversarial example; (iv) object masking, and (v) object-segment
background shifting. Below we provide brief descriptions for these visual
treatments as illustration in Fig. 3.Image Scrambling (IS) (Ye, 2010)
algorithms re-align all pixels in an image to different positions to permute
an original image into a new image, which is used in privacy-preserved
classification (Tarr and Bülthoff, 1998).

Neural Style Transfer (ST) (Gatys et al., 2015) creates texture effect with
perceptual loss (Johnson et al., 2016) and super-resolution along with
instance normalization (Ulyanov et al., 2016).

Adversarial Example (AE) adds input perturbation for prediction evasion. We
employ the Fast Gradient Sign Method (FGSM) (Goodfellow and Jones, 2015) with
a scaled $\ell_{\infty}$ perturbation bound of $\epsilon=0.3$. We also
evaluated other attacks including C&W (Carlini and Wagner, 2017) and PGD
(Madry et al., 2017) in supplement B.

Object Masking (OM) & Background Refilling (BR): Object masking (OM) was
proposed in previous studies (Lopez-Paz et al., 2017; Yang et al., 2019) for
causal learning. We applied OM and another masking methods, background
refilling (BR), that duplicates non-object background into the mask segment as
treatments.

##  5 Experiments

###  5.1 Noisy Image Classification on NICO and CPS

Table 4: Perturbation (e.g., texture) effects with classification accuracy (%)
on the average of CPS images ($\sim$13.7k) for different treatments and their
causal effect estimates. Note that TLT (7.39M) has similar parameters compared
with CVAE’ and CEVAE’, which are enhanced by ResNet as discussion in the
ablation studies. n is for treatment noise level.

Classification Accuracy ($\uparrow$) Average Treatment Effect ($\uparrow$)
Type of t (with n = 0.05) CVAE’ CEVAE’ TLT (ours) CVAE’ CEVAE’ TLT (ours)
Original (without $\mathbf{t}$) 83.31 $\pm 0.12$ 83.31 $\pm 0.23$ 83.31 $\pm
0.13$ 0.012 0.018 0.032 Style Transfer (ST) 73.67 $\pm 0.31$ 74.34 $\pm 0.26$
76.12 $\pm 0.27$ 0.354 0.343 0.318 Image Scrambling (IS) 72.31 $\pm 1.27$
76.21 $\pm 0.81$ 80.12 $\pm 0.54$ 0.057 0.295 0.288 Adversarial Example (AE)
79.12 $\pm 0.25$ 81.12 $\pm 0.17$ 83.12 $\pm 0.12$ 0.025 0.027 0.036 Object
Masking (OM) 70.12 $\pm 0.19$ 72.73 $\pm 0.21$ 73.06 $\pm 0.11$ 0.179 0.241
0.243 Background Refilling (BR) 71.32 $\pm 0.28$ 72.59 $\pm 0.29$ 73.91$\pm
0.17$ 0.213 0.221 0.238

Generative Model Baselines

For a fair comparison, we select two benchmark conditional generative model
incorporating both information of label ($y$) and binary treatment ($t$):
modified conditional VAE (Sohn et al., 2015; Kingma and Welling, 2013) (CVAE’)
and modified CEVAE (Louizos et al., 2017) (CEVAE’), where CVAE’ use $p(t,y)$
for concatenation as a conditional inference and CEVAE’ follows a similar
causal variational inference process (Louizos et al., 2017) without features
fusion and conditional queries. Both model are enhanced by ResNet (He et al.,
2016) and attention layers with similar parameters (7.1M) with TLT. Noted
CEVAE (Louizos et al., 2017) is originally designed and applied only on linear
regression tasks but benefited from our causal modeling for noisy image
classification.

Table 5: Classification accuracy (%) on NICO.

Model StableNet (He et al., 2020) CVAE’ CEVAE’ TLT Acc. 59.76 $\pm 1.52$ 57.23
$\pm 2.12$ 62.17 $\pm 1.82$ 65.98 $\pm 1.74$

Performance on NICO Dataset. We first evaluate models performance trained on
NICO dataset. From the reported results in the paper (He et al., 2020; Zhang
et al., 2021), we select the best reported model, StableNet from (Zhang et
al., 2021) with sample weighting. As shown in Table. 5, generative models with
proposed causal modeling attain competitive results on NICO with compositional
bias setup, where TLT attains a best performance of 65.98%. We provide more
analysis under different setup of NICO, where TLT remains as the best model in
supplement C.

Figure 4: _(a)_ With proposed TLT and CPS dataset, neural saliency methods can
be extended to visual pattern from inference. Take the top row as an example,
using TLT, guided grad-CAM (Selvaraju et al., 2017) can be more aligned with
the concise human-interpretable giraffe patterns instead of forest texture and
edges. More correlation analyses between saliency and labels in NICO and CPS
are given in supplement C. _(b)_ Visualization of learned manifolds of $q(z)$
by tSNE (Maaten and Hinton, 2008), proposed CTR’s results largest intra-
cluster pair-wise sample distances between additive noise (adversarial) ($t$ =
1) and vanilla ($t$ = 0) image samples from CPS1.

Performance on CPS Dataset. In Table 4, we compare TLT with modified CVAE’ and
modified CEVAE’ as baselines trained on CPS dataset. The accuracy of TLT in
the original image, IS, ST, AE, OM and BR settings are consistently better
than CVAE’ and CEVAE’, with substantially large margins ranging from
$\mathbf{1.60\%}$ to $\mathbf{7.81\%}$. CEVAE’ and TLT are also shown to have
higher causal estimate (CE) than CVAE’ in all settings except for ST.
Interestingly, ST leads to a higher causal value (from $\mathbf{0.318}$ to
$\mathbf{0.354}$) when compared to the other modifications such as IS and AT.
This finding accords to the recent studies on DNN’s innate bias of using edges
and textures for vision task (Geirhos et al., 2019). CEVAE’ and TLT having
lower value in ST setting could be explained by a more unbiased representation
learned by inference network with lower dependency on edges and textures. A
benchmark visualization of Guided Grad-CAM (Selvaraju et al., 2017) in Fig. 4
_(a)_ validates this hypothesis and highlights the importance of our inference
network in gaining robust visual understanding from latent space $z$ as tSNE
(Maaten and Hinton, 2008) results Fig. 4 _(6)_. One critical issue for visual
intervention is its difficulty in investigating the effect on object mask size
(Lopez-Paz et al., 2017; Pickup et al., 2014). supplement C shows a consistent
and stable performance of TLT against varying mask sizes.

Case Study on the medical super-class: We conduct the same experiments with
medical super-class to identify visual clinical features. Both the
classification and estimation performance are consistent with general CPS
objects, where TLT attains the highest accuracy $88.74$% in the original
setting and $82.57$% in the scrambling setting (e.g., data encryption
operation) settings. TLT is most effective in classifying noisy image and more
sensible in measuring ATE on adversarial example. We also conduct expert
evaluation on the activation saliency of clinical patterns (Fig. 4). Based on
their domain knowledge (Wang et al., 2017; Rajpurkar et al., 2017a, b), three
physicians independently and unanimously give the highest confidence scores on
saliency attributes to our method.

Statistical Refutation of Causal Models: To rigorously validate our ATE
estimation result, we follow a standard refuting setting (Rothman and
Greenland, 2005; Pearl et al., 2016; Pearl, 1995b) with the causal model in
Fig. 1 to run three major tests, as reported in supplement E and Table S15,
which validate our method is robust.

##  6 Conclusion

Motivated by human-inspired attention mechanism and causal hierarchy theorem,
in this paper we proposed a novel framework named treatment learning
transformer (TLT) for tackling noisy image classification with treatment
estimation. In addition to showing significantly improved accuracy of TLT on
the NICO dataset with noisy contexts, we also curated a new causal-pair
dataset (CPS) based on five different visual image perturbation types for
performance benchmarking on general and medical images. We validated the
causal effect of TLT through statistical refutation testing on average
treatment effects. We also show derived advantages of TLT in terms of improved
visual saliency maps and representation learning. Our results suggest
promising means and a new neural network architecture toward the advancement
of research in the intersection of deep learning and visual causal inference.

## References

  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Pearl (2019) Judea Pearl.  The seven tools of causal inference, with reflections on machine learning.  _Communications of the ACM_ , 62(3):54–60, 2019\.
  * Alcorn et al. (2019) Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen.  Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4845–4854, 2019.
  * Xiao et al. (2015) Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang.  Learning from massive noisy labeled data for image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2691–2699, 2015.
  * Goodfellow et al. (2015) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _ICLR_ , 2015.
  * He et al. (2020) Yue He, Zheyan Shen, and Peng Cui.  Towards non-iid image classification: A dataset and baselines.  _Pattern Recognition_ , page 107383, 2020.
  * Pearl (1995a) Judea Pearl.  Causal diagrams for empirical research.  _Biometrika_ , 82(4):669–688, 1995a.
  * Peters et al. (2014) Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf.  Causal discovery with continuous additive noise models.  _The Journal of Machine Learning Research_ , 15(1):2009–2053, 2014.
  * Louizos et al. (2017) Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling.  Causal effect inference with deep latent-variable models.  In _Advances in Neural Information Processing Systems_ , pages 6446–6456, 2017.
  * Trampusch and Palier (2016) Christine Trampusch and Bruno Palier.  Between x and y: how process tracing contributes to opening the black box of causality.  _New political economy_ , 21(5):437–454, 2016\.
  * Pearl et al. (2016) Judea Pearl, Madelyn Glymour, and Nicholas P Jewell.  _Causal inference in statistics: A primer_.  John Wiley & Sons, 2016.
  * Oquab et al. (2014) Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1717–1724, 2014.
  * Sohn et al. (2015) Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In _Advances in neural information processing systems_ , pages 3483–3491, 2015.
  * Han et al. (2019) Jiangfan Han, Ping Luo, and Xiaogang Wang.  Deep self-learning from noisy labels.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5138–5147, 2019.
  * Li et al. (2017) Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li.  Learning from noisy labels with distillation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1910–1918, 2017.
  * Gal et al. (2017) Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  _arXiv preprint arXiv:1703.02910_ , 2017.
  * Zhang et al. (2021) Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen.  Deep stable learning for out-of-distribution generalization.  _arXiv preprint arXiv:2104.07876_ , 2021.
  * Liu et al. (2021) Jiashuo Liu, Zheyan Shen, Peng Cui, Linjun Zhou, Kun Kuang, Bo Li, and Yishi Lin.  Stable adversarial learning under distributional shifts.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021\.
  * Pickup et al. (2014) Lyndsey C Pickup, Zheng Pan, Donglai Wei, YiChang Shih, Changshui Zhang, Andrew Zisserman, Bernhard Scholkopf, and William T Freeman.  Seeing the arrow of time.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2035–2042, 2014.
  * Fire and Zhu (2016) Amy Fire and Song-Chun Zhu.  Learning perceptual causality from video.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 7(2):23, 2016.
  * Lebeda et al. (2015) Karel Lebeda, Simon Hadfield, and Richard Bowden.  Exploring causal relationships in visual object tracking.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3065–3073, 2015.
  * Fire and Zhu (2013) Amy Fire and Song-Chun Zhu.  Using causal induction in humans to learn and infer causality from video.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 35, 2013.
  * Lopez-Paz et al. (2017) David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou.  Discovering causal signals in images.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6979–6987, 2017.
  * Kocaoglu et al. (2017) Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath.  Causalgan: Learning causal implicit generative models with adversarial training.  _arXiv preprint arXiv:1709.02023_ , 2017.
  * Bahadori et al. (2017) Mohammad Taha Bahadori, Krzysztof Chalupka, Edward Choi, Robert Chen, Walter F Stewart, and Jimeng Sun.  Causal regularization.  _arXiv preprint arXiv:1702.02604_ , 2017.
  * Yang et al. (2019) Chao-Han Huck Yang, Yi-Chieh Liu, Pin-Yu Chen, and Xiaoli Ma.  When causal intervention meets image masking and adversarial perturbation for deep neural networks.  _arXiv preprint arXiv:1902.03380_ , 2019.
  * Tang et al. (2020) Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3716–3725, 2020.
  * Qi et al. (2020) Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang.  Two causal principles for improving visual dialog.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10860–10869, 2020.
  * Agarwal et al. (2020) Vedika Agarwal, Rakshith Shetty, and Mario Fritz.  Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9690–9698, 2020.
  * Abbasnejad et al. (2020) Ehsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi, and Anton van den Hengel.  Counterfactual vision and language learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10044–10054, 2020.
  * Zhang et al. (2020) Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun.  Causal intervention for weakly-supervised semantic segmentation.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Chalupka et al. (2014) Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt.  Visual causal feature learning.  _arXiv preprint arXiv:1412.2309_ , 2014.
  * Bühlmann et al. (2014) Peter Bühlmann, Jonas Peters, Jan Ernest, et al.  Cam: Causal additive models, high-dimensional order search and penalized regression.  _The Annals of Statistics_ , 42(6):2526–2556, 2014.
  * Monti et al. (2020) Ricardo Pio Monti, Kun Zhang, and Aapo Hyvärinen.  Causal discovery with general non-linear relationships using non-linear ica.  In _Uncertainty in Artificial Intelligence_ , pages 186–195. PMLR, 2020.
  * Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _Proceedings of the 31st International Conference on International Conference on Machine Learning-Volume 32_ , pages II–1278, 2014\.
  * Shalit et al. (2017) Uri Shalit, Fredrik D Johansson, and David Sontag.  Estimating individual treatment effect: generalization bounds and algorithms.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 3076–3085. JMLR. org, 2017.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Yang et al. (2021) Chao-Han Huck Yang, I Hung, Te Danny, Yi Ouyang, and Pin-Yu Chen.  Causal inference q-network: Toward resilient reinforcement learning.  _arXiv preprint arXiv:2102.09677_ , 2021.
  * Imbens and Rubin (2010) Guido W Imbens and Donald B Rubin.  Rubin causal model.  In _Microeconometrics_ , pages 229–241. Springer, 2010.
  * Rubin (1974) Donald B Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies.  _Journal of educational Psychology_ , 66(5):688, 1974.
  * Pearl (2009) Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Bareinboim et al. (2020) E Bareinboim, JD Correa, D Ibeling, and T Icard.  On pearl’s hierarchy and the foundations of causal inference.  _ACM Special Volume in Honor of Judea Pearl (provisional title)_ , 2020.
  * Shpitser and Pearl (2008) Ilya Shpitser and Judea Pearl.  Complete identification methods for the causal hierarchy.  _Journal of Machine Learning Research_ , 9(Sep):1941–1979, 2008.
  * Bahuleyan et al. (2018) Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart.  Variational attention for sequence-to-sequence models.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1672–1682, 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Holland (1986) Paul W Holland.  Statistics and causal inference.  _Journal of the American statistical Association_ , 81(396):945–960, 1986.
  * Nauta et al. (2019) Meike Nauta, Doina Bucur, and Christin Seifert.  Causal discovery with attention-based convolutional neural networks.  _Machine Learning and Knowledge Extraction_ , 1(1):312–340, 2019.
  * Chen et al. (2015) Tianwen Chen, Lars Michels, Kaustubh Supekar, John Kochalka, Srikanth Ryali, and Vinod Menon.  Role of the anterior insular cortex in integrative causal signaling during multisensory auditory–visual attention.  _European Journal of Neuroscience_ , 41(2):264–274, 2015.
  * Ramsauer et al. (2020) Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, et al.  Hopfield networks is all you need.  _arXiv preprint arXiv:2008.02217_ , 2020.
  * Greenland et al. (1999) Sander Greenland, James M Robins, Judea Pearl, et al.  Confounding and collapsibility in causal inference.  _Statistical science_ , 14(1):29–46, 1999.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Hill (2011) Jennifer L Hill.  Bayesian nonparametric modeling for causal inference.  _Journal of Computational and Graphical Statistics_ , 20(1):217–240, 2011.
  * LaLonde (1986) Robert J LaLonde.  Evaluating the econometric evaluations of training programs with experimental data.  _The American economic review_ , pages 604–620, 1986.
  * Hoyer et al. (2009) Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf.  Nonlinear causal discovery with additive noise models.  In _Advances in neural information processing systems_ , pages 689–696, 2009.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * Simpson et al. (2019) Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms.  _arXiv preprint arXiv:1902.09063_ , 2019.
  * Reed (2012) Stephen K Reed.  _Cognition: Theories and applications_.  CENGAGE learning, 2012.
  * Musen and Treisman (1990) Gail Musen and Anne Treisman.  Implicit and explicit memory for visual patterns.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ , 16(1):127, 1990.
  * Turk (2012) Amazon Mechanical Turk.  Amazon mechanical turk.  _Retrieved August_ , 17:2012, 2012.
  * Ye (2010) Guodong Ye.  Image scrambling encryption algorithm of pixel bit based on chaos map.  _Pattern Recognition Letters_ , 31(5):347–354, 2010.
  * Tarr and Bülthoff (1998) Michael J Tarr and Heinrich H Bülthoff.  Image-based object recognition in man, monkey and machine.  _Cognition_ , 67(1-2):1–20, 1998.
  * Gatys et al. (2015) Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  A neural algorithm of artistic style.  _arXiv preprint arXiv:1508.06576_ , 2015.
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In _European conference on computer vision_ , pages 694–711. Springer, 2016.
  * Ulyanov et al. (2016) Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  _arXiv preprint arXiv:1607.08022_ , 2016.
  * Goodfellow and Jones (2015) Michael Goodfellow and Amanda L Jones.  Laceyella.  _Bergey’s Manual of Systematics of Archaea and Bacteria_ , pages 1–4, 2015.
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner.  Adversarial examples are not easily detected: Bypassing ten detection methods.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , pages 3–14. ACM, 2017.
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 618–626. IEEE, 2017.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(Nov):2579–2605, 2008.
  * Geirhos et al. (2019) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _ICLR_ , 2019.
  * Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR)_ , pages 3462–3471, 2017.
  * Rajpurkar et al. (2017a) Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  _arXiv preprint arXiv:1711.05225_ , 2017a.
  * Rajpurkar et al. (2017b) Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn L Ball, et al.  Mura: Large dataset for abnormality detection in musculoskeletal radiographs.  _arXiv preprint arXiv:1712.06957_ , 2017b.
  * Rothman and Greenland (2005) Kenneth J Rothman and Sander Greenland.  Causation and causal inference in epidemiology.  _American journal of public health_ , 95(S1):S144–S150, 2005.
  * Pearl (1995b) Judea Pearl.  On the testability of causal models with latent and instrumental variables.  In _Proceedings of the Eleventh conference on Uncertainty in artificial intelligence_ , pages 435–443. Morgan Kaufmann Publishers Inc., 1995b.
  * Albrecht (2016) Jan Philipp Albrecht.  How the gdpr will change the world.  _Eur. Data Prot. L. Rev._ , 2:287, 2016.
  * Zhou et al. (2016) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2921–2929, 2016.
  * Downing et al. (2001) Paul Downing, Jia Liu, and Nancy Kanwisher.  Testing cognitive models of visual attention with fmri and meg.  _Neuropsychologia_ , 39(12):1329–1342, 2001.
  * Luck et al. (2000) Steven J Luck, Geoffrey F Woodman, and Edward K Vogel.  Event-related potential studies of attention.  _Trends in cognitive sciences_ , 4(11):432–440, 2000.
  * Leonards et al. (2000) Ute Leonards, Stefan Sunaert, Paul Van Hecke, and Guy A Orban.  Attention mechanisms in visual search—an fmri study.  _Journal of Cognitive Neuroscience_ , 12(Supplement 2):61–75, 2000.
  * Prakash et al. (2018) Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer.  Deflecting adversarial attacks with pixel deflection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8571–8580, 2018.
  * Ilyas et al. (2019) Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry.  Adversarial examples are not bugs, they are features.  _arXiv preprint arXiv:1905.02175_ , 2019.

##  Appendix A The “What If” Challenges for Deep Neural Networks (DNNs)

A causal inferable DNNs for reasoning chaotic real-world patterns [Pearl,
2009, Yang et al., 2019, Pearl, 1995a] would be necessary for many practice
scenarios. Recent regulatory concerns (e.g., GDPR [Albrecht, 2016]) on
Artificial Intelligence (AI) safety and self-driving automobile accidents also
highlight the importance and the emergence of understanding: (1) "What" does a
DNNs model "learn for accurate label-prediction and (2) utilizing the "Why"
relationship between labels generated by human knowledge and their
conceptional pattern representations in the real world.

We propose COCOCP dataset based on MS-COCO [Lin et al., 2014]. COCOCP dataset
includes classes having similar object(concept) but are different in context
that could possibly result in confusing interpretation and recognition by
human psychology studies (e.g., giraffe and elephant, etc.). In total, three
different common causal pairs are formatted, namely giraffe-elephant (g-e)
with 3316 images, stop sign-fire hydrant (s-f) with 2419 images, and bike-
motorcycle (b-m) with 4729 images. A pair of stop sign and fire hydrant has
been selected to study public awareness of on-road visual detection.

###  A.1 Neural Causation Coefficient (NCC)

Neural Causation Coefficient (NCC) [Lopez-Paz et al., 2017] is a novel
observational causal discovery technique for the joint distribution of a pair
of related proxy variables that are computed by applying CNNs to the image
pixels. NCC leverages such embedded joint distribution as a regularization
term to encourage the learning of causal or anticausal patterns in neural
networks. Lopez et al. [Lopez-Paz et al., 2017] used an augmented NCC network
to prove the existence of causal relations in ResNet [He et al., 2016] between
object and context in an image, and showed that in object-feature ratio
anticausal signal consistently has stronger relation than causal signal.
However, they mainly focus on discussing the object-context causal hypothesis
in the image setting, but barely covers the measurement of perturbation
effect.

(a) TFR calculated by feature fR from ResNet34 [He et al., 2016] as [Lopez-Paz
et al., 2017].

(b) TFR calculated by feature fC from our proposed CAN.

Figure 5: Evaluation of causal pairs by treatment feature ratio (TFR) score
[Lopez-Paz et al., 2017]. The average and standard deviation of TFR associated
to the top-1% causal/anticausal feature scores are displayed. The results show
the visual perturbation measurement is coherent with the previous study
[Lopez-Paz et al., 2017].

From Eq. 12, object-feature ratio (OFR) [Lopez-Paz et al., 2017] could be
extended to a treatment-feature ratio (TFR) score s${}^{t}_{l}$ by do-operator
[Pearl, 2019] as:

$s_{l}^{t}=\frac{\sum_{j=1}^{m}\left|(f_{jl}^{c}|do(t))-(f_{jl}|do(t))\right|}{\sum_{j=1}^{m}\left|(f_{jl)}|do(t)\right|},$
(13)

where ($l$=512) is the input feature length, and f${}^{t}_{j}$ corresponding
vectors of feature scores is equivalent to F${}_{\text{DNN}}$(x${}^{t}_{j}$).

We reproduce the NCC architecture from [Lopez-Paz et al., 2017] and find all
the anti-causal scores of COCO is larger than causal score as shown as [Lopez-
Paz et al., 2017].

###  A.2 Correlation and Causation in Vision Task

Correlation [Pearl, 2009, 2019, 1995a] is a statistical measure that describes
the size and direction of a relationship between two or more variables. A
correlation between variables, however, does not automatically mean that the
change in one variable is the cause of the difference in the values of the
other variable. Causation [Pearl, 2019, 2009] indicates that one event is the
result of the occurrence of the other event; i.e., there is a causal
relationship between the two events. This is also referred to as cause and
effect.

Theoretically, the difference between the two types of relationships are easy
to identify — an action or occurrence can cause another (e.g., having rain
droplet causes an increase in the risk of developing rain day), or it can
correlate with another (e.g., a visual rain droplet is correlated with a red
umbrella, but it does not lead to having a representation of a red umbrella in
vision directly). In practice, however, it remains difficult to establish
cause and effect, compared with establishing correlation clearly. Yet, most of
the current deep learning method focused on directly visualizing the patterns
after the visual model is trained without verifying (e.g., intervention
methods [Pearl, 2019, Yang et al., 2019]) the causation between each
representing patterns.

Figure 6: Generative process of Treatment Inference Transformer model. Solid
lines: decoding phase. Dashed lines: probabilistic encoding phase. The
treatment information could be only observed during training.

###  A.3 Basic Metrics for Causal Inference

By defining the CE of the individual treatment effect (ITE) as the difference
between two potential outcomes for the individual [Louizos et al., 2017,
Shalit et al., 2017], the average treatment effect (ATE) is defined as the
expected value of the potential outcomes over the subjects.

##  Appendix B Dataset

###  B.1 Dataset Statistics: CPS General

In order to accurately measure visual causality, we select six categories from
MS-COCO [Lin et al., 2014] and match them into three causal pairs, which are
giraffe-elephant, bicycle-motorcycle, and stop sign-fire hydrant. Each pair is
chosen by its matching characteristic and background and split into train and
validation set by MS-COCO default setting. With similar characteristics, we
are able to study on the causality of how network classifies different
categories. To avoid network relying on texture to predict, every pair has a
similar background, which makes our classifier more robust to evaluate the
causal effect. Next, we analyze the properties of each pair and compare them.
The number of images and instances per category for both train and validation
set are shown in Fig.7 and Fig.8. In addition, the segmentation size of each
pair is shown in Table.6. The segmentation size of each pair influence the
performance of our classifier for excessive noise from background increasing
the difficulty to find the correct hidden features as shown in Table.7.

(a) Train Set

(b) Validation Set

Figure 7: Image details per pairs in CPS: CPS1 – Giraffe-Elephant, CPS2 –
Bicycle-Motorcycle, and CPS3 – Stop sign-Fire hydrant.

(a) Train Set

(b) Validation Set

Figure 8: Instance details per pairs in CPS: CPS1 – Giraffe-Elephant, CPS2 – Bicycle-Motorcycle, and CPS3 – Stop sign-Fire hydrant. Table 6: Percentage of Segmentation in Images Pair 1 | Giraffe | Elephant
---|---|---
14.60% | 24.13%
Pair 2 | Bicycle | Motorcycle
5.74% | 15.59%
Pair 3 | Stop Sign | Fire Hydrant
7.58% | 7.64%
Table 7: CPS General: performance for common object [Lin et al., 2014] causal pairs with different visual treatments. | Treatment | CVAE’ | CEVAEatt | TIT
---|---|---|---|---
Bicycle & Motorcycle | Object Masking 0.0 | 78.31 $\pm 0.17$ | 80.79 $\pm 0.06$ | 81.05 $\pm 0.11$
Object Masking 0.5 | 74.98 $\pm 0.09$ | 79.46 $\pm 0.12$ | 79.51 $\pm 0.16$
Object Masking 1.0 | 71.65 $\pm 0.23$ | 72.85 $\pm 0.13$ | 73.29 $\pm 0.08$
Background Refilling 0.5 | 75.28 $\pm 0.15$ | 77.5 $\pm 0.27$ | 78.68 $\pm 0.20$
Background Refilling 1.0 | 71.11 $\pm 0.42$ | 74.49 $\pm 0.38$ | 73.95 $\pm 0.41$
Stop Sign & Fire Hydrant | Object Masking 0.0 | 74.59 $\pm 0.29$ | 75.79 $\pm 0.26$ | 77.41 $\pm 0.19$
Object Masking 0.5 | 72.28 $\pm 0.10$ | 73.91$\pm 0.05$ | 74.08 $\pm 0.08$
Object Masking 1.0 | 68.67 $\pm 0.34$ | 71.22 $\pm 0.28$ | 71.06 $\pm 0.24$
Background Refilling 0.5 | 69.13 $\pm 0.16$ | 73.79 $\pm 0.21$ | 75.45 $\pm 0.14$
Background Refilling 1.0 | 65.62 $\pm 0.47$ | 66.65 $\pm 0.37$ | 68.24 $\pm 0.44$
Elephant & Giraffe | Object Masking 0.0 | 93.72 $\pm 0.25$ | 93.53 $\pm 0.28$ | 94.67 $\pm 0.20$
Object Masking 0.5 | 90.14 $\pm 0.11$ | 93.01 $\pm 0.19$ | 93.15 $\pm 0.09$
Object Masking 1.0 | 80.12 $\pm 0.19$ | 82.73 $\pm 0.21$ | 83.06 $\pm 0.11$
Background Refilling 0.5 | 90.44 $\pm 0.08$ | 91.71 $\pm 0.11$ | 91.73 $\pm 0.10$
Background Refilling 1.0 | 81.32 $\pm 0.28$ | 82.59 $\pm 0.29$ | 83.91 $\pm 0.17$

###  B.2 Dataset Statistics: CPS Medical

In total, 2,633 three-dimensional images (with 658 test images) were collected
across multiple anatomies of interest, multiple modalities, and multiple
sources (or institutions) representative of real-world clinical applications
followed by COCO-CP processing. All images were identified using processes
consistent with institutional review board polices at each contributing site.
We reformatted the images to reduce the need for specialized software packages
for reading to encourage use by specialists in medical imaging for high-level
feature reasoning.

Human Evaluation
Chest MRI can provide important features to diagnose lung problems such as a
tumor or pleural disorder, blood vessel problems, or abnormal lymph nodes. We
collaborate with three board-certified thoracic surgeons to review the
activate region generated by guided grad-CAM [Selvaraju et al., 2017] on the
test images. The surgeons individually retrospectively reviewed and labeled
each study from the generated 100 image results as a DICOM file as consistent
or inconsistent saliency compared with their diagnosis using the PACS system.
The radiologists have averaged 6.43 years of experience on average, ranging
from 5 to 16 years.

The TIT-generated saliency results also attain the highest consistency (61.2%)
from thoracic surgeons compared with the results from VAERes (48.1%) and
CEVAE${}^{*}_{Res}$ (58.8%). The consistency from a randomly generated
saliency map is only (3.2%).

###  B.3 Ablation Study

Starting from a ResNet, we modified the architecture towards proposed TIT and
compare the accuracy performance of various architecture. Table. 9 shows the
impact of each change of the architecture on COCO${}_{C}P$ classification.
Among all variation, attention mechanism is the most important feature, while
having bilinear fusion (BF) is also more effective than concatenation.

Figure 9: We use class activation mapping methods [Zhou et al., 2016, Selvaraju et al., 2017] to explain our medical classification model. The yellow bounding box is ground truth label from the Decathlon [Simpson et al., 2019] dataset. The guided-grad CAM method shows a highest false-negative scores on the region of interest.  Figure 10: We show Guided-Grad-CAM results on different classification model. The results generated from CAM is much matching yellow bounding box from the Decathlon [Simpson et al., 2019] dataset in the test dataset.  Table 8: Accuracy performance under adversarial attack as the treatment. TIT attains higher accuracy in both FGSM, C&W, and PGD settings.  Method | CVAE’ | CEVAE’ | TIT
---|---|---|---
FGSM [Goodfellow and Jones, 2015] | 91.92$\pm 0.11$ | 92.02$\pm 0.11$ | 92.86$\pm 0.12$
C&W [Carlini and Wagner, 2017] | 82.32 $\pm 2.34$ | 74.23$\pm 4.18$ | 88.12$\pm 1.26$
PGD [Madry et al., 2017] | 74.32 $\pm 1.38$ | 86.34$\pm 1.70$ | 89.43$\pm 1.08$

##  Appendix C Parameter and Architecture

###  C.1 Adversarial Perturbation

With recent security concerns of adversarial example over visual recognition,
we also made a broad study on the accuracy and causal effect under adversarial
examples. Fast Gradient Sign Method (FGSM) [Goodfellow and Jones, 2015] is a
classical gradient-based adversarial noise to generate adversarial examples by
one step gradient update along the direction of the sign of gradient at each
pixel by:

$X_{\text{Adversarial}}=X+\varepsilon\cdot\operatorname{sign}\left(\nabla_{X}J(X,Y)\right),$
(14)

where $J$ is the training loss (e.g. cross entropy) and $Y$ is the groundtruth
label for $X$. We adopt FGSM as an visual modification with $\varepsilon$ =
$\mathbf{0.3}$ $\ell_{\infty}$ perturbation constraint. This treatment could
be further extended on other adversarial examples combined with causal
analysis [Yang et al., 2019]. Instead of FGSM, we also study the accuracy
performance under Carlini-Wagner attack (C&W) [Carlini and Wagner, 2017] and
projected gradient descent (PGD) [Madry et al., 2017] as treatment. As shown
in Table 8, our proposed TIT attains higher accuracy and less accuracy
degradation in FGSM, C&W and PGD settings compared to CVAE’ and CEVAE’ for CPS
classification.

###  C.2 Overparameterization

For a fair comparison, we study the performance of architectures with similar
number of parameters. To align with the number of parameter in TIT, We modify
the number of Resblocks in CVAE’ as 4 and add attention mechanism to CEVAE’.
As shown in Table 10, with similar number of parameters, our proposed TIT
acquire the highest accuracy and better utilize the power of more parameters
to compete with the state-of-art CVAE’ architecture.

###  C.3 Different Mask Size

We study the effect of different mask sizes with same intervention flipping
rate. The object-masking and background-refilling are used as visual
perturbation in the experiments. To observe the effect, we gradually increased
the mask ratio among the target object. The results in Table 12 and Table 14
show the impact of changing the ratio to the accuracy of CPS general and
medical dataset classification. We find the accuracy drops as the ratio
increasing, while our proposed TIT is relatively resilient to the high noise
ratio scenario and perform better classification.

###  C.4 NICO Dataset Settings

NICO dataset provides several settings to simulate the Non-I.I.D dataset on
different levels. 4 typical settings to generate Non-I.I.D training and
testing subset.
Minimum Bias
The setting choose the images in target class as positive samples and images
in other classes as negative samples ignoring the context, which could lead to
a minimum distribution shift in training and testing subset.
Proportional Bias
The setting takes all context into consideration but the ratio of each context
are different in training and testing subset. In this setting, the level of
distribution shift can be adjusted based on the difference of context ratio.
Compositional Bias
In this setting, the contexts exist in test subset are not guaranteed to exist
in training subset. The distribution shift is higher between training and
testing set. The shift could be enhanced by adding proportional bias.

We carefully observe the effect of different settings imposing on different
model. The result in Table. 11 shows our proposed TIT performs better in all
settings compared to other models including the CNBB model proposed by [He et
al., 2020].

Table 9: Model architecture ablation study in COCOCP Architecture | Val. Acc. (%)
---|---
ResNet | 81.23$\pm 0.12$
ResNet + CVAE = CVAE’ | 82.31$\pm 0.13$
ResNet + CEVAE = CEVAE” | 82.17$\pm 0.24$
CEVAE + BF - bernoulli = CEVAE’ | 82.68$\pm 0.15$
Treatment Inference Transformer (TIT) | 84.32$\pm 0.07$
Table 10: Overparameterization ablation study in COCOCP Model | Para. | Val. Acc. (%)
---|---|---
CVAE’ | 4.03M | 82.31$\pm 0.13$
CVAE’ + 2 Resblocks | 5.92M | 82.38 $\pm 0.18$
CVAE’ + 4 Resblocks | 7.83M | 81.96 $\pm 0.19$
CEVAE’ + AttentionC | 7.81M | 83.62 $\pm 0.21$
TIT (ours) | 7.39M | 84.92$\pm 0.07$
Figure 11: Error bar of average treatment effect Table 11: Classification accuracy (%) on NICO dataset with different model and setting. Setting 1,2 and 3 refers to minimum bis, proportional bias and compositional bias mentioned in C separately. Setting |  CNBB [He et al., 2020] | CVAE’ | CEVAE’ | TIT
---|---|---|---|---
Setting 1 | 42.96 $\pm 1.54$ | 48.72 $\pm 2.01$ | 53.94 $\pm 1.74$ |  57.02 $\pm 1.42$
Setting 2 | 44.15 $\pm 1.48$ | 50.10 $\pm 1.98$ | 54.33 $\pm 1.64$ |  58.75 $\pm 1.44$
Setting 3 | 45.16 $\pm 1.52$ | 50.23 $\pm 2.12$ | 56.17 $\pm 1.82$ |  60.98 $\pm 1.52$
Table 12: Accuracy (%) of varying mask sizes when intervention flipping rate
$\mathbf{n}=0.05$. $\mathbf{I_{OM}}$/$\mathbf{I_{BR}}$ denotes Object-
Masking/Background-Refilling. The number $X\%$ means masking $X\%$ of a target
object. Our proposed method (TIT) maintains relatively high accuracy as $X$
increases.

$\mathbf{I_{OM}}$ CVAE’ CEVAEatt TIT $\mathbf{I_{BR}}$ CVAE CEVAEatt TIT 10%
93.31 $\pm 0.17$ 92.58 $\pm 0.21$ 94.32$\pm 0.08$ 10% 93.04 $\pm 0.16$ 94.25
$\pm 0.18$ 94.65 $\pm 0.09$ 30% 91.19 $\pm 0.15$ 93.37 $\pm 0.19$ 94.13$\pm
0.07$ 30% 91.27 $\pm 0.21$ 93.53 $\pm 0.23$ 94.25 $\pm 0.11$ 50% 90.14 $\pm
0.11$ 93.01 $\pm 0.19$ 93.15 $\pm 0.09$ 50% 90.44 $\pm 0.08$ 91.71 $\pm 0.11$
91.73 $\pm 0.10$ 70% 86.53 $\pm 0.13$ 91.90 $\pm 0.23$ 91.26 $\pm 0.18$ 70%
86.62 $\pm 0.21$ 88.85 $\pm 0.18$ 90.46  $\pm 0.12$ 100% 80.12 $\pm 0.19$
82.73 $\pm 0.21$ 83.06  $\pm 0.11$ 100% 81.32 $\pm 0.28$ 82.59 $\pm 0.29$
83.91 $\pm 0.17$

##  Appendix D Reproducibility

###  D.1 Hyper-Parameters and Experiment Setup

Causal Effect Autoencoder[Louizos et al., 2017] (CEVAE${}^{*}_{Res}$)
baseline: To empower CEVAE for the visual data, our input images use dim(C)=3,
dim(X)=128, dim(Y)=128. The encoder part of VAE model utilized in paper takes
the ResNet34 as feature extractor. Then we sample the $q(t|x)$ by Bernoulli
distribution, and $q(y|x,t)$ and $q(z|x,y,t)$ are sampled by densely connected
hidden layer of 512 neurons. Sequentially, the $Z$ is generated by
reparameterization from $q|t$. The decoder starts from 3 ResBlocks with 512
width for the dim(Z)=512 to reconstruct the $p(x|z)$ , we further used 5
upsample blocks with 2 times scaling up and convolution layers with
[512,256,128,64,32] width. For the last convolution layer, we use reflection
padding with width set as 3. Also, we sample the $p(t|z)$ and $p(y|t,z)$ by
projecting t and $\mu_{y}(t)$ through adaptive pooling and the densely
connected hidden layer with 512 width.

CEVAE with Attention baseline (CEVAEAtt): For a much fair comparison with
proposed Treatment Inference Transformer (TIT), we apply the dual attention
module in the encoder part of CEVAE to approximate the $q(z|x,t,y)$. The dual
attention module consists of position and channel attention module.

  * •

Global feature: After 1x1 convolution, the input is scaled by 4 times larger
with bilinear interpolation.

  * •

Position attention: The module outputs the position attention combined by 2
convolution layers with 64 width to calculate the attention.

  * •

Channel attention: The module outputs the channel attention by fusing the
channel into spatial information and pass the feature to 2 convolution layers
with 64 width.

  * •

Combination: The output has the channel of 512, the same width as the input.

The weights of the network are initialized with weights from a model pre-
trained on ImageNet. The Adam algorithm with standard parameters and learning
rate 0.001 are utilized for optimization. We use mini-batches of size 128 and
pick the models with the highest accuracy. All experiments of our model are
implemented in PyTorch using an NVIDIA GeForce GTX 2080 Ti GPU with 12GB
memory. The training time for each MS-COCO [Lin et al., 2014] causal pair with
different visual treatment takes one hour to two hours on average. The
reproducible code of CAN networks and a causal graphical model have been
provided in the supplementary and will be open source111Please follow the
readme in the supplementary open-source code for more information.

###  D.2 Cognitive Response to Attention Mechanism

Cognitive psychology and neuroimaging [Downing et al., 2001] studies have
found a distinct neural response to the different visual scene, as the visual
attention mechanism [Luck et al., 2000]. Attention exercises, Luck et al.,
[Luck et al., 2000] have been proved to be enhanced learning capacities by
executive control and transferring to cognitive abilities. Since images from
different categories vary systematically in their visual properties as well as
their semantic category, variation in visual property may influence our
cognitive process of visual stimuli. The human brain has the ability to
distinguish the visual scenes from different categories when categorical
perception is impaired. For example, scrambling and masking are used widely
when experimenting with visual pattern sensitivity. Although these
perturbation preserved many of their visual characteristics, perception of
scene categories was severely impaired, which makes scrambling and masking
suitable metrics to compare the visual perception process between neural
network and the human brain. These experiments [Luck et al., 2000, Leonards et
al., 2000] have been validate on adding attention training for a improved
learning performance in human education.

###  D.3 Using Saliency Map to Associate Learned Causal Patterns

To better understand the learned causal patterns from TIT, we use class
activation mapping [Zhou et al., 2016] (CAM) to study [Yang et al., 2019,
Pearl, 2019, 2009, 1995a] the causal patterns. CAM removes all fully-connected
layers at the end, and including a tensor product (followed by softmax), which
takes as input the global-average-pooled convolutional feature maps, and
outputs the probability for each class. To obtain the class-discriminative
localization map, Grad-CAM computes the gradient of $\mathbf{yc}$ (score for
class $\mathbf{c}$) with respect to feature maps $\mathbf{A}$ and importance
weights $\mathbf{\alpha_{k}^{c}}$ of a convolutional layer. Similar to CAM,
Grad-CAM [Selvaraju et al., 2017] heat-map is a weighted combination of
feature maps, and followed by a ReLU:

$L_{\mathrm{Grad}\mathrm{CAM}}^{c}=\operatorname{Re}LU\left(\sum_{k}\alpha_{k}^{c}A^{k}\right)$
(15)

In our DNN visualization experiment, we use the state-of-the-are CAM method,
guide-GradCAM [Selvaraju et al., 2017] for comparing CVAE’, CEVAE’, and our
TIT. guided-GradCAM fuse guided backpropagation and the Grad-CAM
visualizations via a point-wise multiplication.

Interestingly, according to the intervened image after class-activation
mapping techniques in Fig. 4 in the main context and Fig. 12 in the
supplementary, we could find out when the area of interest are much central on
the texture and edge effect. To reduce the texture dependent variable, we
utilize a neural style transfer on the image set before the intervention.

Table 13: Refuting tests of the causal estimate [Pearl, 2019] with causal effect (CE) over different treatment. The validation tests show our method is confident since the common random selection (Tc) and Subset test (Ts) are closed to the original CE, and all the CE results after replacing treatment with a random (placebo) variable (Tp) are close to zero. Treatment | Original ATE |  Test-Common (Tc) $\uparrow$ | Test-Placebo (Tp) $\downarrow$ | Test-Subset (Ts) $\uparrow$
---|---|---|---|---
IS: TIT | 0.288 | 0.288 | 0.00479 | 0.288
CEVAEatt | 0.2948 | 0.2941637 | 0.0427 | 0.276
CVAE’ | 0.057 | 0.05673101 | 0.0385 | 0.0583
AT: TIT | 0.036 | 0.035 | 0.012 | 0.035
CEVAEatt | 0.027 | 0.0274 | 0.0062 | 0.024
CVAE’ | 0.0247 | 0.0242 | 0.01347 | 0.0156
SB: TIT | 0.2334 | 0.23385 | 0.0253 | 0.238
CEVAEatt | 0.2417 | 0.2431 | 0.0364 | 0.2353
CVAE | 0.1853368 | 0.1853 | 0.01157 | 0.1834
IM: TIT | 0.1855 | 0.1854 | 0.037 | 0.191
CEVAEatt | 0.22 | 0.22 | 0.0038 | 0.1736
CVAE’ | 0.222 | 0.22285 | 0.0200707 | 0.1609
ST: TIT | 0.31763 | 0.317641 | 0.0278 | 0.3351
CEVAEatt | 0.3431 | 0.342221 | 0.0225 | 0.3252
CVAE’ | 0.354412 | 0.354334 | 0.01127 | 0.3257
Table 14: Classification accuracy (%) with error bars (with 10-fold cross-validation) comparison between different visual treatments under intervention in the CPS dataset.  Treatment | CVAE’ | CEVAEatt | TIT
---|---|---|---
Object Masking 0.0 | 93.61 $\pm 0.15$ | 93.31$\pm 0.11$ | 94.91$\pm 0.15$
Object Masking 0.1 | 93.31 $\pm 0.17$ | 93.58$\pm 0.21$ |  94.32 $\pm 0.08$
Object Masking 0.3 | 91.19 $\pm 0.15$ | 93.37$\pm 0.19$ |  94.13 $\pm 0.07$
Object Masking 0.5 | 90.14 $\pm 0.11$ | 93.01 $\pm 0.19$ |  93.15 $\pm 0.09$
Object Masking 0.7 | 86.53 $\pm 0.13$ | 91.90$\pm 0.23$ | 91.26$\pm 0.18$
Object Masking 1.0 | 80.12 $\pm 0.19$ | 82.73 $\pm 0.21$ |  83.06 $\pm 0.11$
Background Refilling 0.1 | 93.04 $\pm 0.16$ | 94.25 $\pm 0.18$ |  94.65 $\pm 0.09$
Background Refilling 0.3 | 91.27 $\pm 0.21$ | 93.53 $\pm 0.23$ |  94.25 $\pm 0.11$
Background Refilling 0.5 | 90.44 $\pm 0.08$ | 91.71 $\pm 0.11$ | 91.75$\pm 0.10$
Background Refilling 0.7 | 86.62 $\pm 0.21$ | 88.85 $\pm 0.18$ |  90.46 $\pm 0.12$
Background Refilling 1.0 | 81.32 $\pm 0.28$ | 82.59 $\pm 0.29$ |  83.91 $\pm 0.17$
Image Scrambling | 59.42 $\pm 2.19$ | 77.3 $\pm 1.17$ |  78.8 $\pm 0.62$
Style Transfer | 67.73 $\pm 2.19$ | 68.12 $\pm 1.21$ |  68.29 $\pm 0.42$
Adversarial Example | 91.92 $\pm 0.11$ | 92.02 $\pm 0.11$ |  92.86 $\pm 0.12$
Figure 12: We conduct activation saliency experiment on the COCO-CP trained by
bike patterns under partial masking intervention. The guided GradCAM results
show current DNNs methods often overfit on the texture and background patterns
instead of desired training label(s), which echoes to previous studies
[Prakash et al., 2018, Yang et al., 2019, Ilyas et al., 2019].

##  Appendix E Identification of Visual Causal Effect

###  E.1 Causality

Rubin’s Causal Model (Sekhon, 2008) is a framework developed for the
statistical analysis of cause of effect based on the idea of potential
outcomes. Consider:
$\bullet~{}t_{i},$ a binary treatment $\mathbf{t}$ for individual $i$ with 1
referring to assigning the treatment and 0 to no treatment;
$\bullet~{}y_{i}$ is the outcome on individual $i$ given a treatment value.
Each individual can have two potential outcomes or (counterfactuals) available
as $\\{y_{i}(1),y_{i}(0)\\}$ $\mathbf{t}$ corresponding to receiving the
treatment or not.

Identifying conceptional treatment effect Individual Treatment Effect (ITE)
can be defined as the difference between the two potential outcomes for the
individual; Average Treatment Effect (ATE) as the expected value of the
potential outcomes over the subjects. For a binary outcome, it is defined as:
given by:

$y_{i}=y_{i}(0)\left(1-t_{i}\right)+y_{i}(1)t_{i};\hskip
2.84544ptATE=\mathbb{E}\left[y_{i}(1)\right]-\mathbb{E}\left[y_{i}(0)\right];\hskip
2.84544pt$ (16)

$\displaystyle y_{i}=y_{i}(0)\left(1-t_{i}\right)+y_{i}(1)t_{i};$
$\displaystyle
ATE=\mathbb{E}\left[y_{i}(1)\right]-\mathbb{E}\left[y_{i}(0)\right];\hskip
2.84544pt$ (17)

The above mentioned metric cannot be properly estimated if there are
confounding variables in the system, which will introduce bias (Greenland et
al., 1999). The causal effect by a treatment variable t on an outcome y is
represented by $E[y|do(t=1)]$, where do represents the fact that the treatment
has been kept at a specific value by external interventions on the system
which do not affect other variables and their causal relationships in the
system. Pearl defines the causal effect for a given treatment t and an outcome
y and other confounding variables $\mathbf{Z}$ as:

Given a proxy X, outcome y, binary treatment t and confounder Z, we use the
back-door criteria to get:

$\displaystyle P(y|X,do(t=1))=$
$\displaystyle\int_{Z}P(y|X,do(t=1),Z)P(Z|X,do(t=1))dZ$ (18)

Using the intervention manipulation rules, we obtain:

$P(y|X,do(t=1))=\int_{Z}P(y|X,t=1,Z)P(Z|X)dZ.$ (19)

The refuting test for all conditional visual model show sustainable
performance to the original ATE by random common cause variable test (Tc) and
random subset test (Ts) and an ideally nearby zero ATE results on replacing
treatment (Tr) with a random variable test. Above validation show our CGM and
its associated neural are robust and validated for causal modeling and
measurement.

Table 15: Validation of causal effect by three causal refuting tests. The causal effect estimate is tested by random common cause variable test (Tc), replacing treatment with a random (placebo) variable (Tr – lower is better), and removing a random subset of data (Ts). TIT outperforms in most tests. Noise : do(t) | Measurement of ATE
---|---
Method | Original | w/ Tc | w/ Tp | w/ Ts
TIT | 0.2432 | 0.2431 | 0.0114 | 0.2481
CEVAE’ | 0.2414 | 0.2414 | 0.0248 | 0.2329
CVAE’ | 0.1792 | 0.1763 | 0.0120 | 0.1751

##  Appendix F Evidence Lower Bound of VAE

To validate an Evidence Lower Bound (ELBO) of our CAN, we assume
$\mathbf{p(X,Z)}$, where $\mathbf{X}$ is the observed data and $\mathbf{Z}$ is
the latent representation. $\mathbf{p(X,Z)}$ can be decomposed into the
likelihood and the prior as: $\mathbf{p(X,Z)}$ = $\mathbf{p(X|Z)p(Z)}$. Using
Baye’s inference to calculate the posterior gives:

$p(Z|X)=\frac{p(X|Z)p(Z)}{\int_{z}p(X|z)p(z)}$ (20)

VAE approximates it with the family of distributions $q_{\lambda}(Z|X)$, where
$\lambda$ is the variational of parameters for the given family. We minimize
the $KL$ divergence to ensure that the approximate distribution used is close
to the true posterior:

$\displaystyle KL\left(q_{\lambda}(Z|X)\|p(X|Z)\right)=$
$\displaystyle\mathbb{E}_{q}\left[\log\left(q_{\lambda}(Z|X)\right)\right]$
(21) $\displaystyle-\mathbb{E}_{q}[\log p(X,Z)]+\log p(X).$

The posterior for inference network will be :

$q_{\lambda}^{*}(Z|X)=\underset{\lambda}{\arg\min}KL\left(q_{\lambda}(Z|X)\|p(X|Z)\right).$
(22)

However, due to the occurrence of $\mathbf{p(X)}$, the KL is still
intractable. We can manipulate the above equation by defining the ELBO:

$\displaystyle
ELBO(\lambda)=\log(p(X))-KL\left(q_{\lambda}(Z|X)\|p(X|Z)\right)$
$\displaystyle=\mathbb{E}_{q}[\log p(X|Z)]-KL\left(\log
q_{\lambda}(Z|X)\|p(Z)\right).$ (23)

Then, the negative of the ELBO is the loss function used for the neural
networks:

$\displaystyle l(\theta,\phi)=$
$\displaystyle-\mathbb{E}_{q_{\theta}(z|x)}\left[\log p_{\phi}(X|Z)\right]$
(24) $\displaystyle+KL\left(\log q_{\theta}(Z|X,\lambda)||p(Z)\right).$

$\theta$ and $\phi$, are the weights and biases of the DNN which are chosen to
maximize the ELBO using gradient descent algorithm.
Training Objective of Treatment Inference Transformer. In the TIT setting,
where the architecture is adapted from TARnet [Shalit et al., 2017]’s
inference network, i.e., split input for each treatment group in t after a
shared representation, the objective function $\mathcal{L}$ is given by:

$\displaystyle\mathcal{L}=\sum_{i=1}^{N}\mathbb{E}_{q\left(\mathbf{z}_{i}|\mathbf{x}_{i},t_{i},y_{i}\right)}\left[\log
p\left(\mathbf{x}_{i},t_{i}|\mathbf{z}_{i}\right)+\log
p\left(y_{i}|t_{i},\mathbf{z}_{i}\right)\right]$
$\displaystyle+\sum_{i=1}^{N}\mathbb{E}_{q\left(\mathbf{z}_{i}|\mathbf{x}_{i},t_{i},y_{i}\right)}[\log
p\left(\mathbf{z}_{i}\right)-\log
q\left(\mathbf{z}_{i}|\mathbf{x}_{i},t_{i},y_{i}\right)]$ (25)

For predicting new subject predictions, the treatment assignment $t$ along
with outcome $y$ are required. We have introduced Bernoulli distributions
which help predict $y$ and $t$ (a binary index of treatment) for new samples
with the theoretical foundation from CEVAE [Louizos et al., 2017]. We then
leverage bilinear fusion for $q(z|x,y,t,a)$ instead of concatenation [Louizos
et al., 2017] and remove Bernoulli sampling for classification label
inference. The attention decoding $p(a|x,q(y))=q(a)$ is incorporating with the
known treatment for training.

##  Appendix G Future Work

Discovering Visual Causality beyond Vision Classification Tasks.
In conclusion, we find out causal effect do exist in different DNN-based
visual modification methods, and this effect could be visualized to see its
effectiveness on understanding targeted DNN layer. By introducing a new
extended dataset, COCO-CPs, our CAN networks show competitive visualization
results and potential combined with existing saliency-based methods. For
future work, we plan to extend our proposed CAN framework to discover visual
causality over more visual tasks, such a video detection, cross-model
adaption, and obvious question answering (VQA).