# Causal Attention for Vision-Language Tasks

Xu Yang1 Hanwang Zhang1 Guojun Qi2 Jianfei Cai3
1School of Computer Science and Engineering  Nanyang Technological University
Singapore
2Futurewei Technologies
3Faculty of Information Technology  Monash University  Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a novel attention mechanism: Causal Attention (CATT), to remove the
ever-elusive confounding effect in existing attention-based vision-language
models. This effect causes harmful bias that misleads the attention module to
focus on the spurious correlations in training data, damaging the model
generalization. As the confounder is unobserved in general, we use the front-
door adjustment to realize the causal intervention, which does not require any
knowledge on the confounder. Specifically, CATT is implemented as a
combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention
(CS-ATT), where the latter forcibly brings other samples into every IS-ATT,
mimicking the causal intervention. CATT abides by the Q-K-V convention and
hence can replace any attention module such as top-down attention and self-
attention in Transformers. CATT improves various popular attention-based
vision-language models by considerable margins. In particular, we show that
CATT has great potential in large-scale pre-training, e.g., it can promote the
lighter LXMERT [61], which uses fewer data and less computational power,
comparable to the heavier UNITER [14]. Code is published in
https://github.com/yangxuntu/catt.

##  1 Introduction

Stemming from the strong cognitive evidences in selective signal processing
[64, 54], the attention mechanism has arguably become the most indispensable
module in vision and language models [71, 5, 3, 16, 11, 39]. Although its
idiosyncratic formulation varies from task to task, its nature can be
summarized as the following common Q-K-V notation: given a _query_
$\mathbf{q}$, the attention mechanism associates $\mathbf{q}$ to each feature
_value_ $\mathbf{v}_{i}$ by using the normalized attentive weight
$\alpha_{i}\propto\mathbf{q}^{T}\mathbf{k}_{i}$, where $\mathbf{k}_{i}$ is the
_key_ function of the value; thus, the resultant selective feature value —
attention — is $\sum_{i}\alpha_{i}\mathbf{v}_{i}$. In a modern view, the
attention can be understood as a feature _transformer_ that encodes input
query $\mathbf{q}$ by using the given values $\bm{V}=\\{\mathbf{v}_{i}\\}$
[65].

Figure 1: Top: an example of image captioner with a self-attention and a top-
down attention modules. Bottom: the corresponding causal graph. The reason why
the prediction is “riding” but not “driving” is explained in Figure 3.

Taking image captioning as an example in Figure 1, if $\mathbf{q}$ and
$\bm{V}$ are both encoded from the input $X$, , the RoI features of an image,
we call it _self-attention_ ; if $\mathbf{q}$ is changed to the sentence
context, we call it _top-down attention_. Intuitively, self-attention is
usually viewed as a non-local [70] (or graph [7]) convolution network that
enriches each local value with global relationship features; top down-
attention is used to enrich the context with the cross-domain relationship
features [3]. Both of them can be combined and stacked into deep networks,
serving as powerful multi-modal encoder-decoder transformer networks [35, 12].

As a bridge connecting the input feature $X$ and the output label $Y$, the
quality of attention — how reasonable the attentive weight $\alpha$ is — plays
a crucial role for the overall performance. However, due to the fact that the
attention weights are unsupervised, , there is no word-region grounding for
the top-down attention or relationship dependency annotation for the self-
attention, the weights will be inevitably misled by the dataset bias. For
example, as shown in Figure 1, since there are many images captioned with
“person riding horse” in the training data, self-attention learns to infer
“riding” by building the dependency between “person” and “horse”. Then, given
a test image with “person driving carriage”, this self-attention still tends
to relate “person” with “horse” to infer “riding”, but ignoring the
“carriage”. Unfortunately, such bias cannot be mitigated by simply enlarging
the dataset scale, as most of the bias abides by the data nature — Zipf’s law
[51] and social conventions [19] — there are indeed more “red apple” than
“green apple” or “person standing” than “person dancing”. Therefore, as shown
in Figure 2, large-scale pre-training may lead to even worse attentions.

Figure 2: Before pre-training (, LXMERT [61]), attentions are correct (blue).
After pre-training, attentions are wrong (red). This is because the co-
occurrences of some concepts appear much more often than others, , “Sport+Man”
appears 213 times more than “Sport+Screen” in the pre-training data.

Figure 3: This expands the causal links of the confounding path
$X\dashleftarrow\dashrightarrow Y$ in Figure 1

.

The dataset bias is essentially caused by the confounder, a common cause that
makes $X$ and $Y$ correlated even if $X$ and $Y$ have no direct causation. We
illustrate this crucial idea in Figure 3. Suppose that the confounder $C$ is
the common sense111It is also well-known as the disentangled causal mechanism
[60]. “person can ride horse”, $C\rightarrow X$ denotes that a visual scene is
generated by such knowledge, , the dataset curator observes and captures the
common sense; $X\rightarrow M$ denotes the fact that the objects
$M=\\{\text{person},\text{horse}\\}$ can be detected (, Faster R-CNN [52]),
whose object inventory is determined by $C\rightarrow M$; $M\rightarrow Y$
denotes the language generation for “person riding horse”. Note that besides
the legitimate causal path from image $X$ via object $M$ to $Y$, the
“backdoor” path $X\leftarrow C\rightarrow M\rightarrow Y$ also contributes an
effect to $Y$. Therefore, if we only train the model based on the correlation
$P(Y|X)$ without knowing the confounding effect, no matter how large the
amount of training data is, the model can never identify the true causal
effect from $X$ to $Y$ [44, 56]. For example, if the confounder distribution
varies from training to testing, , the common sense “person can ride horse” is
dominantly more often than the common sense “person can drive carriage” in
training, but the latter is more often than the former in testing, then
$P(Y|X)$ based on “person can ride horse” in training will be no longer
applicable in testing [45].

In this paper, we propose a novel attention mechanism called: _Causal
Attention_ (CATT), which can help the models identify the causal effect
between $X$ and $Y$, and thus mitigates the bias caused by confounders. It is
based on the _front-door adjustment_ principle that does not require the
assumption of any observed confounder [43], and thus CATT can be applied in
any domain where the attention resides. In this way, CATT is fundamentally
different from existing deconfounding methods based on the backdoor adjustment
[83, 69], which has to be domain-specific to comply with the observed-
confounder assumption. Specifically, we first show that the conventional
attention is indeed an improper approximation of the front-door principle, and
then we show what is a proper one, which underpins CATT theoretically (Section
3.1).

We build CATT on the proposed In-Sample attention (IS-ATT) and Cross-Sample
attention (CS-ATT), which abides by the Q-K-V operations (Section 3.2). In
particular, the parameters of the Q-K-V operations can also be shared between
both IS-ATT and CS-ATT to further improve the efficiency in some
architectures. We replace the the conventional attention with CATT in various
vision-language models to validate its effectiveness, including the classic
Bottom-Up Top-Down LSTM [3], Transformer [65], and a large-scale vision-
language pre-training (VLP) model LXMERT [61]. The experimental results
demonstrate that our CATT can achieve consistent improvements for all of them.
Significantly, our light LXMERT+CATT outperforms the heavy UNITER [14] on
VQA2.0, , 73.04% vs. 72.91% on test-std split, and NLVR2, , 76.0% vs. 75.80%
on test-P split, while we require much fewer pre-training burdens: 624 vs. 882
V100 GPU hours. Such comparisons show that our CATT has great potential in
vision-language pre-training (VLP) tasks.

##  2 Related Work

Attention Mechanism. Conventional top-down attentions generally include the
classic single-guidance fashion [5, 71, 75, 77] and the co-guidance fashion
[37, 79]. They can be summarized as the query, key, value (Q-K-V) operation
that also generalizes to self-attention [65, 70], which even be applied in
pure vision tasks such as visual recognition and generation [11, 12]. As the
attention weight is unsupervised, it is easily misled by the confounders
hidden in the dataset. We exploit the causal inference to propose a novel CATT
module to mitigate the confounding effect [47, 44]. As our proposed CATT
complies with the Q-K-V convention, it has great potential in any model that
uses attention.

Vision-Language Pre-Training. Inspired by the success of large-scale pre-
training for language modeling [16, 50], researchers have developed some
multi-modal Transformer-based Vision-Language Pre-training (VLP) models to
learn task-agnostic visiolinguistic representations [35, 61, 31, 14, 85, 32,
30]. To discover the visiolinguistic relations across domains, a huge amount
of data [57, 13, 28] are required for VLP. However, just as the language pre-
training models tend to learn or even amplify the dataset bias [29, 40], these
VLP models may also overplay the spurious correlation. We use the proposed
CATT to help VLP models confront the bias.

Causal Inference. Causality [44, 56] provides researchers new methodologies to
design robust measurements [60], discover hidden causal structures [9],
generate counterfactual samples [62, 1, 27, 80], and confront various biases
[67, 81, 83, 41, 20, 48]. These bias removal methods usually assume that the
confounder is observable [81, 83] or domain-specific [19, 10]. In general, the
confounder is unobservable and elusive. Compared with them, we exploit the
front-door adjustment [43] with no observed-confounder assumption to mitigate
the dataset bias. To tackle the sampling challenge in the front-door
adjustment, we propose two effective approximations called In-Sample Sampling
and Cross-Sample Sampling.

##  3 Causal Attention

###  3.1 Attention in the Front-Door Causal Graph

We retrospect the attention mechanism in a front-door causal graph [47, 44] as
shown in the bottom part of Figure 1, where the causal effect is passed from
the input set $X$ to the target $Y$ through a mediator $Z$. By this graph, we
can split the attention mechanism into two parts: a selector which selects
suitable knowledge $Z$ from $X$ and a predictor which exploits $Z$ to predict
$Y$. Take VQA as the example, $X$ is a multi-modality set containing an image
and a question, then the attention system will choose a few regions from the
image based on the question to predict the answer. We usually use the
observational correlation $P(Y|X)$ as the target to train an attention-based
model:

$\small P(Y|X)=\underbrace{\sum\nolimits_{z}P(Z=z|X)}_{\text{IS-
Sampling}}{P(Y|Z=z)},$ (1)

where $z$ denotes the selected knowledge and IS-Sampling denotes In-Sample
sampling since $z$ comes from the current input sample $X$.

However, as discussed in Introduction, since the selection is an unsupervised
process, the predictor may be misled by the dataset bias when training it by
Eq. (1). In causal terms, this means that the predictor may learn the spurious
correlation brought by the backdoor path $Z\leftarrow X\leftrightarrow
Y$111For convenience, we simplify the notation of the backdoor path
$X\leftarrow C\rightarrow M\rightarrow Y$ shown in Figure 3 to
$X\leftrightarrow Y$. instead of the true causal effect $Z\rightarrow Y$, and
thus the conventional attention mechanism is not a proper way of calculating
the causal effect.

To eliminate the spurious correlation brought by the hidden confounders, we
should block the backdoor path between $Z$ and $Y$: $Z\leftarrow
X\leftrightarrow Y$. In this way, we can estimate the true causal effect
between $Z$ and $Y$, which is denoted as $P(Y|do(Z))$, where $do(\cdot)$
denotes the interventional operation [44]. We can cut off the link
$X\rightarrow Z$ to block this backdoor path by stratifying the input variable
$X$ into different cases $\\{\bm{x}\\}$ and then measuring the average causal
effects of $Z$ on $Y$ by the following expectation [46]:

$\small P(Y|do(Z))=\underbrace{\sum\nolimits_{x}P(X=x)}_{\text{CS-Sampling
}}P(Y|X=x,Z),$ (2)

where $x$ denotes one possible input case. Here we denote it as Cross-Sample
Sampling (CS-Sampling) since it comes from the other samples. Intuitively, CS-
Sampling approximates the “physical intervention” which can break the spurious
correlation caused by the hidden confounder. For example, the annotation “man-
with-snowboard” is dominant in captioning dataset [19] and thus the predictor
may learn the spurious correlation between the snowboard region with the word
“man” without looking at the person region to reason what actually the gender
is. CS-Sampling alleviates such spurious correlation by combining the person
region with the other objects from other samples, , bike, mirror, or brush,
and inputting the combinations to the predictor. Then the predictor will not
always see “man-with-snowboard” but see “man” with the other distinctive
objects and thus it will be forced to infer the word “man” from the person
region. With this deconfounded predictor, the selector will also be forced to
select the legitimate evidence even we do not have any region-word
supervisions.

By replacing $P(Y|\bm{z})$ in Eq. (1) by $P(Y|do(Z))$ in Eq. (2), we can
calculate the true causal effect between $X$ and $Y$:

$\displaystyle P(Y|do(X))$ (3) $\displaystyle=$
$\displaystyle\underbrace{\sum\nolimits_{z}P(Z=z|X)}_{\text{IS-
Sampling}}\underbrace{\sum\nolimits_{x}P(X=x)}_{\text{CS-Sampling
}}[P(Y|Z=z,X=x)].$

This is also called the front-door adjustment, which is a fundamental causal
inference technique for deconfounding the unobserved confounder [43]. Since
our novel attention module is designed by using Eq. (3) as the training
target, we name our attention module as Causal Attention (CATT).

###  3.2 In-Sample and Cross-Sample Attentions

To implement our causal attention (Eq. (3)) in a deep framework, we can
parameterize the predictive distribution $P(Y|Z,X)$ as a network $g(\cdot)$
followed by a softmax layer since most vision-language tasks are transformed
into classification formulations [68, 4]:

$\displaystyle P(Y|Z,X)=\text{Softmax}[g(Z,X)].$ (4)

As can be seen in Eq. (3), we need to sample $X$ and $Z$, and feed them into
the network to complete $P(Y|do(X))$. However, the cost of the network forward
pass for all of these samples is prohibitively expensive. To address this
challenge, we apply Normalized Weighted Geometric Mean (NWGM) approximation
[71, 58] to absorb the outer sampling into the feature level and thus only
need to forward the “absorbed input” in the network for once. Specifically, by
NWGM approximation, IS-Sampling and CS-Sampling in Eq. (3) can be absorbed
into the network as:

$\displaystyle P(Y|do(X))\approx$
$\displaystyle\text{Softmax}[g(\hat{\bm{Z}},\hat{\bm{X}})],$ (5)
$\displaystyle\textbf{IS-Sampling:}\quad\hat{\bm{Z}}=$
$\displaystyle\sum\nolimits_{z}P(Z=z|h(X))\bm{z},$ $\displaystyle\textbf{CS-
Sampling:}\quad\hat{\bm{X}}=$
$\displaystyle\sum\nolimits_{x}P(X=x|f(X))\bm{x}.$

where $h(\cdot)$ and $f(\cdot)$ denote query embedding functions which can
transform the input $X$ into two query sets. Both of them can be parameterized
as networks. Note that in a network, the variable $X$ and $Z$ are represented
by embedding vectors, , an image region becomes an RoI representation, so we
use bold symbols to signify these embedding vectors, , $\bm{z}$, $\bm{x}$
denote the embedding vectors of the variable $z$, $x$. $\hat{\bm{X}}$,
$\hat{\bm{Z}}$ denote the estimations of the IS-Sampling and CS-Sampling,
which can be packed into the matrix form [65]. The derivation details of Eq.
(5) are given in the supplementary material.

Actually, the IS-Sampling estimation $\hat{\bm{Z}}$ is what a classic
attention network calculates, which can be briefly expressed by the Q-K-V
operation as the blue block in Figure 4:

Input: $\displaystyle\bm{Q}_{I},\bm{K}_{I},\bm{V}_{I},$ (6) Prob:
$\displaystyle\bm{A}_{I}=\text{Softmax}({\bm{Q}_{I}}^{T}{\bm{K}_{I}})$ Ouput:
$\displaystyle\hat{\bm{Z}}=\bm{V}_{I}\bm{A}_{I}$

We denote Eq. (6) as the In-Sample attention (IS-ATT) and the subscript “$I$”
emphasizes that it is estimating IS-Sampling. In this case, all the
$\bm{K}_{I}$ and $\bm{V}_{I}$ come from the current input sample, , the RoI
feature set. $\bm{Q}_{I}$ comes from $h(X)$, , in top-down attention, the
query vector $\bm{q}_{I}$ is the embedding of the sentence context and in
self-attention, the query set $\bm{Q}_{I}$ is also the RoI feature set. For
$\bm{A}_{I}$, each attention vector $\bm{a}_{I}$ is the network estimation of
the IS-Sampling probability $P(Z=z|h(X))$ and the output $\hat{\bm{Z}}$ is the
estimated vector set of IS-Sampling in Eq. (5).

Inspired by Eq (6), we can also deploy a Q-K-V operation to estimate
$\hat{\bm{X}}$ and name it as Cross-Sample attention (CS-ATT), which is the
red block in Figure 4:

Input: $\displaystyle\bm{Q}_{C},\bm{K}_{C},\bm{V}_{C},$ (7) Prob:
$\displaystyle\bm{A}_{C}=\text{Softmax}({\bm{Q}_{C}}^{T}{\bm{K}_{C}}),$ Ouput:
$\displaystyle\hat{\bm{X}}=\bm{V}_{C}\bm{A}_{C}$

where $\bm{K}_{C}$, $\bm{V}_{C}$ come from the other samples in the training
set, and $\bm{Q}_{C}$ comes from $f(X)$. In this case, $\bm{a}_{C}$
approximates $P(X=x|f(X))$ and $\hat{\bm{X}}$ is the CS-Sampling estimation in
Eq. (5). In the implementations, we set $\bm{K}_{C}$ and $\bm{V}_{C}$ as the
global dictionaries compressed from the whole training dataset since it is
impossible to attend to all the samples in the training set. Specifically, we
initialize this dictionary by using K-means over all the samples’ embeddings
in training set, , all the images’ RoI features. In this way, $\bm{V}_{C}$ and
$\bm{V}_{I}$ stay in the same representation space, which guarantees that the
estimations of IS-Sampling and CS-Sampling: $\hat{\bm{Z}}$ and $\hat{\bm{X}}$
in Eq. (5) also have the same distribution.

Figure 4:  The sketch of a single causal attention module, which includes an
IS-ATT (Eq. (6)) and a CS-ATT (Eq. (7)). After calculating $\hat{\bm{Z}}$ and
$\hat{\bm{X}}$, we can input them into the predictor for making decisions or
more stacked attention layers for further embedding.

To sum up, as shown in Figure 4, our single causal attention module estimates
$\hat{\bm{Z}}$ and $\hat{\bm{X}}$ respectively by IS-ATT in Eq. (6) and CS-ATT
in Eq. (7). After that, we can concatenate the outputs for estimating
$P(Y|do(X))$ as in Eq. (5).

###  3.3 CATT in Stacked Attention Networks

In practice, attention modules can be stacked as deep networks, , the classic
Transformer [65] or BERT architectures [16]. Our CATT can also be incorporated
into these stacked attention networks and we experiment with Transformer [65]
and LXMERT [61] in this paper. We briefly introduce their architectures here
and discuss the implementation details in Section 4.2. Generally, our CATT
replaces the first attention layer of these architectures to get the
estimations of IS-Sampling $\hat{\bm{Z}}$ and CS-Sampling $\hat{\bm{X}}$, and
then we input them into more attention layers for further embedding, as shown
in Figure 4. For convenience, in these stacked attention networks, we still
use IS-ATT and CS-ATT as the names of the attention modules to signify that
this attention layer is dealing with the representations of the IS-Sampling or
CS-Sampling.

Figure 5: The Transformer+CATT architecture, which contains a visual encoder
and a language decoder. We only show two layers in both parts for
demonstrating how they are connected. In the implementations, both the encoder
and decoder contain 6 layers. $[\bm{V}_{I}]_{E}$ and $[\bm{V}_{C}]_{E}$ denote
the IS-ATT and CS-ATT outputs of the encoder, which are used as the inputs to
the decoder. $\hat{\bm{Z}}$ and $\hat{\bm{X}}$ are the IS-ATT and CS-ATT
outputs of the decoder, which are the estimations of IS-Sampling and CS-
Sampling, respectively.

Transformer+CATT. Figure 5 shows the architecture of our vision-language
Transformer+CATT. This architecture contains a vision encoder and a language
decoder. In implementations, both the encoder and decoder contain 6 blue and
purple blocks. The inputs of the encoder include the embedding set of the
current image and a global image embedding dictionary. The IS-ATT and CS-ATT
outputs of the encoder are input into the decoder for learning visiolinguistic
representations. For the decoder, the inputs of the first IS-ATT and CS-ATT
are respectively the current sentence embedding set and a global sentence
embedding dictionary. The outputs of the decoder include two parts which
respectively correspond to IS-Sampling $\hat{\bm{Z}}$ and CS-Sampling
$\hat{\bm{X}}$, which will be concatenated and input into the final predictor.
Importantly, by stacking many CATT layers, the estimated $\hat{\bm{Z}}$ and
$\hat{\bm{X}}$ may not stay in the same representation space due to the non-
convex operations in each attention module, , the position-wise feed-forward
Networks [65]. To avoid this, we share the parameters of IS-ATT and CS-ATT in
each CATT and then the outputs of them will always stay in the same
representation space, where the detail formations are given in Eq. (8). As a
result, the additional attention computation of CATT in LXMERT is
$O(K*n)$/$O(n*n)$ at the first/other layer, where $K$ is the size of the
global dictionary and $n$ is the number of word/image sequence.

LXMERT+CATT. Figure 6 demonstrates the architecture of our LXMERT+CATT, which
contains three parts, a vision encoder with 5 self-CATT modules, a language
encoder with 9 self-CATT modules, and a visiolinguistic decoder with 5 blocks
where each one contains two cross-modality CATT (CM-CATT) and two self-CATT
modules. For convenience, we merge the inputs (outputs) of IS-ATT and CS-ATT
into one single line in (c). For example, the image inputs contain two parts
which are the current image and a global image embedding dictionary.
$[\bm{V}]_{V}$ ($[\bm{V}]_{L}$) denotes the visual (language) signal which
also includes two parts $[\bm{V}_{I}]_{V}$ and $[\bm{V}_{C}]_{V}$
($[\bm{V}_{I}]_{L}$ and $[\bm{V}_{C}]_{L}$) corresponding to IS-ATT and CS-
ATT, respectively. Figure 6(b) sketches one cross-modality module used in the
top part of the decoder in (c), where the visual signals are used as the
queries in both IS-ATT and CS-ATT. Similar as the original LXMERT [61], we
concatenate the outputs of both vision and language streams and input them
into various predictors for solving different vision-language tasks. In
implementations, we share the parameters of IS-ATT and CS-ATT in each causal
attention module to force their outputs to have the same distributions.

Figure 6: The architecture of LXMERT+CATT, which contains one visual encoder,
one language encoder, and one visiolinguistic decoder. Note that each line in
(c) contains two parts which respectively correspond to IS-ATT and CS-ATT.
$[\bm{V}]_{V}$ and $[\bm{V}]_{L}$ denote the visual and language signals,
respectively.

##  4 Experiments

We validated our Causal Attention (CATT) in three architectures for various
vision-language tasks: Bottom-Up Top-Down (BUTD) LSTM [3] for Image Captioning
(IC) [13, 38] and Visual Question Answering (VQA) [4], Transformer [65] for IC
and VQA, and a large scale vision-language pre-training framework LXMERT [61]
for VQA, Graph Question Answering (GQA) [22], and Natural Language for Visual
Reasoning (NLVR) [59].

###  4.1 Datasets

MS COCO [13] has 123,287 images and each image is assigned with 5 captions.
This dataset has two popular splits: the Karpathy split [24] and the official
test split, which divide the whole dataset into $113,287/5,000/5,000$ and
$82,783/40,504/40,775$ for training/validation/test, respectively. We used the
Karpathy split to train the BUTD and Transformer based captioners and
evaluate.

VQA2.0 [18] collects the images from MS COCO and assigns 3 questions for each
image and 10 answers for each question. There are 80k/40k training/validation
images available offline. We exploited the training set to train our BUTD and
Transformer based VQA systems, and then evaluated the performances on three
different splits: offline validation, online test-development, and online
test-standard.

Pre-training and Fine-tuning Datasets for VLP. We followed LXMERT [61] to
collect a large-scale vision-language pre-training dataset from the training
and development sets of MS COCO, VQA2.0, GQA [22], and Visual Genome [28].
After collecting, this dataset contained 180K distinct images and 9.18M image-
sentence pairs. We fine-tuned our VLP model on three tasks, which were VQA,
GQA, and NLVR2 [59] and evaluated the performances on various test splits of
them.

###  4.2 Implementation Details

The implementation details of BUTD+CATT and Transformer+CATT are given in C.4
of the supplementary material. Here we provide the details of LXMERT+CATT,
which is the most significant experiments in this paper.

LXMERT + CATT. We used the architecture in Figure. 6 for large-scale vision-
language pre-training. In this architecture, all IS-ATT and CS-ATT were
deployed by 12-head scaled dot-product [65]:

Input: $\displaystyle\bm{Q},\bm{K},\bm{V}$ (8) Prob:
$\displaystyle\bm{A}_{i}=\text{Softmax}(\frac{\bm{Q}\bm{W}_{i}^{Q}(\bm{K}\bm{W}_{i}^{K})^{T}}{\sqrt{d}})$
$\displaystyle\textbf{Single-Head}:$
$\displaystyle\bm{H}_{i}=\bm{A}_{i}\bm{V}\bm{W}_{i}^{V},$ Ouput:
$\displaystyle\hat{\bm{V}}=\text{Embed}([\bm{H}_{1},...,\bm{H}_{1}2]\bm{W}^{H}),$

where $\bm{W}_{i}^{*}$ and $\bm{W}^{H}$ are all trainable matrices;
$\bm{A}_{i}$ is the soft attention matrix for the $i$-th head; $[\cdot]$
denotes the concatenation operation, and Embed$(\cdot)$ means the feed-forward
network and the residual operation as in [65]. The hidden size was set to 768.
Importantly, we shared the parameters between IS-ATT and CS-ATT in each CATT
to make the outputs stay in the same representation space. In this case, we
also applied K-means to get the initializations and set the size of both
dictionaries to 500. We extracted 36 RoI object features from each image by a
Faster-RCNN [52] pre-trained on VG as in [3].

We followed the original LXMERT [61] to pre-train our LXMERT+CATT architecture
by four tasks: masked cross-modality language modeling, masked object
prediction, cross-modality image sentence matching, and image question
answering. We used Adam optimizer with a linear-decayed learning rate schedule
[16] where the peak learning rate was set to $5e^{-5}$. We pre-trained the
model 20 epochs on 4 GTX 1080 Ti with a batch size of 192. The pre-training
cost 10 days. To fairly compare the pre-training GPU hours with UNITER [14],
we also carried an experiment by 4 V100 with batch size as 256 and it cost 6.5
days for pre-training. When fine-tuning the pre-trained model on VQA2.0, GQA,
and NLVR2, the batch size was 32, training epochs was 4, and the learning
rates were set to $5e^{-5}$, $1e^{-6}$, and $5e^{-5}$, respectively.

###  4.3 Results and Analysis.

####  4.3.1 Image Captioning (IC)

Table 1: The performances of various captioners on Karpathy split.

Models | B@4 | M | R | C | S
---|---|---|---|---|---
BUTD [3] | $37.2$ | $27.5$ | $57.3$ | $125.3$ | $21.1$
LBPF [49] | $38.3$ | $\bm{28.5}$ | $58.4$ | $127.6$ | $22.0$
GCN-LSTM [76] | $38.2$ | $\bm{28.5}$ | $58.3$ | $127.6$ | $22.0$
SGAE [73] | $38.4$ | $28.4$ | $\bm{58.6}$ | $127.8$ | $\bm{22.1}$
BUTD+CATT | $\bm{38.6}$ | $\bm{28.5}$ | $\bm{58.6}$ | $\bm{128.3}$ | $21.9$
Transformer | $38.6$ | $28.5$ | $58.4$ | $128.5$ | $22.0$
VLP [85] | $39.5$ | $-$ | $-$ | $129.3$ | $\bm{23.2}$
AoANet [21] | $38.9$ | $29.2$ | $58.8$ | $129.8$ | $22.4$
$\mathcal{M}^{2}$Transformer [15] | $39.1$ | $29.2$ | $58.6$ | $131.2$ | $22.6$
Transformer+CATT | $39.4$ | $\bm{29.3}$ | $\bm{58.9}$ | $\bm{131.7}$ | $22.8$

Similarity Measurements. The results are reported in Table 1, where the top
and bottom parts list various models which respectively deploy LSTM and
Transformer as the backbones. In this table, B, M, R, C, and S denote BLEU
[42], METEOR[6], ROUGE [33], CIDEr-D [66], and SPICE [2], respectively, which
evaluate the similarities between the generated and the ground-truth captions.

Compared with two baselines BUTD and Transformer, we can find that BUTD+CATT
and Transformer+CATT respectively achieve 3.0-point and 3.2-point improvements
on CIDEr-D. More importantly, after incorporating our CATT modules into BUTD
and Transformer, they have higher CIDEr-D scores than certain state-of-the-art
captioners which deploy more complex techniques. For example, SGAE exploits
scene graphs to transfer language inductive bias or
$\mathcal{M}^{2}$Transformer learns multi-level visual relations though
additional meshed memory networks. These comparisons suggest that our CATT
module is a more powerful technique compared with the techniques used in these
state-of-the-art captioners.

Table 2: The bias degree of different models: “$\uparrow$” and “$\downarrow$”
mean the higher the better and the lower the better, respectively. Red numbers
denote the improvements after using our CATT modules.

Models | CHs$\downarrow$ | CHi$\downarrow$ | A@Gen$\uparrow$ | A@Attr$\uparrow$ | A@Act$\uparrow$
---|---|---|---|---|---
BUTD | 13.5 | 8.9 | 77% | 41% | 52%
BUTD+CATT | 10.7-2.8 | 7.2-1.7 | 85%+8% | 51%+10% | 60%+8%
Transformer | 12.1 | 8.1 | 82% | 47% | 55%
Transformer+CATT | $9.7_{\color[rgb]{1,0,0}{-2.4}}$ | $6.5_{\color[rgb]{1,0,0}{-1.6}}$ | $92\%_{\color[rgb]{1,0,0}{+10\%}}$ | $56\%_{\color[rgb]{1,0,0}{+9\%}}$ | $64\%_{\color[rgb]{1,0,0}{+9\%}}$

Figure 7: Six examples show that our CATT can correct different models to
focus on the suitable regions. TF denotes Transformer. For BUTD, we show the
region with the highest attention weight. For Transformer and VLP, the red
region has the highest attention weight in top-down attention and the green
region is the one most related to the red region in self-attention.  Table 3:
Accuracy of various VQA models on different splits.

Models | loc-val | test-dev | test-std
---|---|---|---
BUTD [3] | 63.20 | 65.32 | 65.67
MUTAN [8] | - | 66.01 | 66.38
MLB [25] | 65.07 | 66.27 | 66.62
BUTD+CATT | $\bm{65.12}$ | $\bm{67.13}$ | $\bm{67.26}$
Transformer | 66.29 | 69.53 | 69.82
DFAF [17] | 66.21 | 70.22 | 70.34
MCAN [79] | 67.20 | 70.63 | 70.90
TRRNet [72] | - | 70.80 | 71.20
Transformer+CATT | $\bm{67.33}$ | $\bm{70.95}$ | $\bm{71.27}$

Table 4: Accuracy of different question types on test-std split. Red numbers
denote the improvements after using our CATT modules.

Models | Yes/No | Number | Other
---|---|---|---
BUTD | 81.82 | 44.21 | 56.05
BUTD+CATT | 83.42+1.6 | 48.96+4.75 | 57.3+1.25
Transformer | 86.25 | 50.7 | 59.9
Transformer+CATT | 87.40+1.15 | 53.45+2.75 | 61.3+1.4
LXMERT† | 88.17 | 52.63 | 62.73
LXMERT+CATT | 88.6+0.43 | 55.48+2.85 | 63.39+0.66

Table 5: Training burdens and performances of different large-scale vision-
language pre-training models. “M” denotes million.

Models | Training Burdens | VQA2.0 | GQA | NLVR2 (Pair)
---|---|---|---|---
GPU Hours | Image / Text | test-dev | test-std | test-dev | test-std | loc-val | test-P
LXMERT [61] | 960 (titan xp) | 0.18M / 9.18M | 72.42 | 72.54 | 60.00 | 60.30 | 74.9 | 74.5
LXMERT† [61] | 816 (1080Ti) | 0.18M / 9.18M | 71.96 | 72.18 | 59.90 | 59.94 | 74.8 | 74.4
ERNIE-VIL [78] | - | 4.20M / 9.58M | 72.62 | 72.85 | - | - | - | -
UNITER [14] | 882 (V100) | 4.20M / 9.58M | 72.80 | 72.91 | - | - | 75.85 | 75.80
12IN1 [36] | 960 (V100) | 5.40M / 7.48M | - | 72.92 | 60.48 | - | - | -
LXMERT†+CATT | 960 (1080Ti), 624 (V100) | 0.18M / 9.18M | $72.81$ | $73.04$ | $60.84$ | $61.17$ | $76.40$ | $76.00$
LXMERT†+CATT$\uparrow$ | 1536 (1080Ti), 1056 (V100) | 0.18M / 9.18M | $\bm{73.54}$ | $\bm{73.63}$ | $\bm{61.87}$ | $\bm{62.07}$ | $\bm{77.27}$ | $\bm{77.23}$

Bias Measurements. We measured the bias degree of the generated captions in
Table 2 to validate that whether our CATT module can alleviate the dataset
bias or not. In this table, CHs and CHi denote CHAIRs and CHAIRi [55], which
are designed to measure the object bias. Apart from them, we also analyze
three more specific biases: gender bias, action bias, and attribute bias by
calculating the accuracy of these words, which are denoted as A@Gen, A@Attr,
and A@Act, respectively. From the results, we can see that after incorporating
our CATT module, both BUTD and Transformer generate less biased captions, ,
the accuracies of gender, attribute, and action are respectively improved by
10%, 9%, and 9% when CATT is used in Transformer. The first row of Figure 7
shows two examples where BUTD and Transformer respectively attend to
unsuitable regions and then generate incorrect captions, , BUTD attend to the
remote region and infer the word “man” due to the dataset bias, while our CATT
corrects this by attending to the hand region to generate the word “hand”.

####  4.3.2 Visual Question Answering (VQA)

The top and bottom parts of Table 3 respectively report the performances of
various LSTM and Transformer based VQA models, where loc-val, test-dev, and
test-std denote the offline local validation, online test-development, and
online test-standard splits. From this table, we can observe that after
deploying our CATT module into BUTD and Transformer, the accuracies are
consistently improved. More importantly, the deconfounded BUTD and Transformer
outperform certain state-of-the-art models which are better than the original
BUTD and Transformer.

Table 4 reports the accuracies of different question types on test-std split.
It can be found that the accuracy of number has the largest improvements after
using CATT modules, , 4.75-point and 2.75-point for BUTD and Transformer,
respectively. Significantly, Transformer+CATT has a higher number accuracy
than the large-scale pre-training model LXMERT: 53.45 vs. 52.63. As analyzed
in [84], the counting ability depends heavily on the quality of the attention
mechanism that a VQA model cannot correctly answer number questions without
attending to all the queried objects. Thus the consistent improvements in
number support that our CATT modules can largely ameliorate the quality of the
conventional attention mechanism. The second row of Figure 7 shows that after
incorporating CATT, BUTD and Transformer based VQA models can attend to the
right regions for answering the questions.

####  4.3.3 Vision-Language Pre-training (VLP)

Table 5 shows the training burdens and the performances of various large-scale
pre-training models on VQA2.0, GQA, and NLVR2. Note that LXMERT† and LXMERT
respectively denote the results got from the officially released code and from
the published paper. For ERNIE-VIL [78] and UNITER [14], they both have a BASE
version and a LARGE version where BASE (LARGE) uses 12 (16) heads and 768
(1024) hidden units in multi-head product operations. We report the
performances of their BASE versions since our model used 12 heads and 768
hidden units. For NLVR2, we report the performances of UNITER with the same
Pair setting as our model.222The details of NLVR2 setting can be found in
Table 5 of UNITER [14].

From this table, we can see that compared with LXMERT†, our LXMERT†+CATT
respectively achieves 0.86, 1.23, 1.6-point improvements on the test-std
splits of VQA2.0 and GQA and the test-P split of NLVR2. For example, compared
with UNITER which uses fp16, our LXMERT†+CATT uses fewer GPU hours and the
pre-training data, while we have higher performances on VQA2.0: 73.04 vs.
72.91, and NLVR2: 76.0 vs. 75.80. Furthermore, inspired by UNITER [14], we
enhanced our model and named this one as LXMERT+CATT$\bm{\uparrow}$ by using
conditional masking and more RoI features. Specifically, we extracted 64 RoI
features from each image to guarantee that our model can be trained on 4 1080
Ti GPUs. It can be found that after using two insights from UNITER, our
LXMERT+CATT$\bm{\uparrow}$ can achieve higher performances than UNITER, even
though we do not extract 100 RoI features for each image as them. These
comparisons suggest that our CATT has great potential in large-scale VLP.

Also, as shown in Table 4, after incorporating CATT into LXMERT, we can
observe that the accuracy of Number is further improved: 55.48 vs. 52.63,
which suggests that our CATT improves the quality of the attention modules in
VLP models. The third row of Figure 7 shows two examples where CATT modules
correct LXMERT to focus on the right regions for answering the questions.

###  4.4 Ablation Studies

We carried exhaustive ablation studies to validate three variants of our
causal attention module: K-means initialization, dictionary size, parameter
sharing. In particular, we deployed these ablation studies in Transformer+CATT
and LXMERT+CATT architectures.

Comparing Methods. Base: We denote the original Transformer and LXMERT
architectures as Base. CATT w/o Init: We use CATT to denote the architectures
which deploy the CATT modules as in Section 3.3. We did not use the K-means
algorithm to initialize the global dictionaries but randomly initialized them.
We shared the parameters between IS-ATT and CS-ATT in these models. CATT w/o
Share: We did not share the parameters between IS-ATT and CS-ATT. Here we used
the K-means algorithm to initialize the dictionaries. CATT+D#K: We set the
size of the global image and word embedding dictionaries to $K$ by the K-means
algorithm and shared the parameters between IS-ATT and CS-ATT.

Table 6: The performances of various CATT ablation studies on the local
validation sets. IC denotes Image Captioning and we report the CIDEr-D score
of these IC methods. For the other tasks, we report their accuracies. The left
and right parts respectively show the ablation studies of the Transformer and
LXMERT architectures.

Models | Transformer | LXMERT
---|---|---
IC | VQA | VQA | GQA | NLVR2
Base | 128.5 | 66.29 | 69.52 | 59.82 | 74.80
CATT w/o Init | 129.8 | 66.56 | 69.81 | 60.14 | 75.22
CATT w/o Share | 130.6 | 66.94 | 70.05 | 60.41 | 75.78
CATT+D#100 | 131.1 | 67.02 | 70.12 | 60.62 | 75.94
CATT+D#200 | 131.4 | 67.21 | 70.29 | 60.77 | 76.26
CATT+D#500 | $\bm{131.7}$ | $\bm{67.33}$ | $\bm{70.40}$ | $\bm{60.90}$ | $\bm{76.40}$

Results and Analysis. Table 6 reports the performances of the ablation studies
on the local validation sets of different tasks. Firstly, we can observe that
after using our CATT architecture, even without K-means initialization or
parameter sharing, the performances are better than Base models. Also, we can
observe that both K-means initialization and parameter sharing are useful for
improving the performances. For example, in LXMERT+CATT, after using K-means
and sharing the parameters, the performances of VQA are respectively boosted:
70.40 vs. 69.81 and 70.40 vs. 70.05. Such observation suggests that both
strategies encourage the estimated IS-Sampling and CS-Sampling to stay in the
same representation space, which is indeed beneficial in improving the
performances. Also, by comparing the performances with different dictionary
sizes, we can find that bigger dictionaries have better performances.

##  5 Conclusion

In this paper, we exploited the causal inference to analyze why the attention
mechanism is easily misled by the dataset bias and then attend to unsuitable
regions. We discovered that the attention mechanism is an improper
approximation of the front-door adjustment and thus fails to capture the true
causal effect between the input and target. Then a novel attention mechanism:
causal attention (CATT) was proposed based on the front-door adjustment, which
can improve the quality of the attention mechanism by alleviating the ever-
elusive confounding effect. Specifically, CATT contains In-Sample and Cross-
Sample attentions to estimate In-Sample and Cross-Sample samplings in the
front-door adjustment and both of two attention networks abide by the Q-K-V
operations. We implemented CATT into various popular attention-based vision-
language models and the experimental results demonstrate that it can improve
these models by considerable margins. In particular, CATT can promote a light
VLP model comparable to a heavy one, which demonstrates its great potential in
large-scale pre-training.

This supplementary document will further detail the following aspects in the
submitted manuscript: A. Causal Preliminaries, B. Formula Derivations, C. More
results D. Implementation Details.

##  6 Causal Preliminaries

###  6.1 Structural Causal Model

Figure 8: Two Structural Causal Models which are (a) a backdoor model and (b)
a front-door model.

In Causality [46, 47], a Structural Causal Model (SCM) is used to describe the
causal relationships. Such a graph connects different variables by directed
edges which denote the causal directions. For example, as shown in Figure
8(a), $X\rightarrow Y$ denotes that $X$ is the cause of $Y$. In an SCM, if a
variable is the common cause of two variables, it is called the confounder.
For example, $C$ is the cause of both $X$ and $Y$, thus it is a confounder
which will induce spurious correlation between $X$ and $Y$ to disturb the
recognition of the causal effect between them. In particular, such spurious
correlation is brought by the backdoor path created by the confounder.
Formally, a backdoor path between $X$ and $Y$ is defined as any path from $X$
to $Y$ that starts with an arrow pointing into $X$. For example, in Figure
8(a), the path $X\leftarrow C\rightarrow Y$ is a backdoor path. Here we use
another two examples for helping understand this concept, as in Figure 8(b),
$X\leftarrow C\rightarrow Y\leftarrow Z$ and $Z\leftarrow X\leftarrow
C\rightarrow Y$ are two backdoor paths between $X$ and $Z$ and $Z$ and $Y$,
respectively.

In an SCM, if we want to deconfound two variables $X$ and $Y$ to calculate the
true causal effect, we should block every backdoor path between them [47]. For
example, in Figure 8(a), we should block $X\leftarrow C\rightarrow Y$ to get
the causal effect between $X$ and $Y$.

###  6.2 Blocking Three Junctions

In an SCM, there are three elemental “junctions” which construct the whole
graph and we have some basic rules to block them. In particular, three
junctions are given as follows:

1\. $X\rightarrow Z\rightarrow Y$. This is called chain junction, which
constructs a front-door path between $X$ and $Y$, as shown in Figure 8(b). In
this junction, once we know the value of the mediator $Z$, learning about $X$
will not give us any information to raise or lower our belief about $Y$. Thus,
if we know what $Z$ is or directly intervene it as a specific value, we block
this chain junction.

2\. $X\leftarrow C\rightarrow Y$. This is called confounding junction which
induces spurious correlation between $X$ and $Y$, as shown in Figure 8(a). In
this junction, once we know what the value of $C$ is or directly intervene it
to a specific value, there is no spurious correlation between $X$ and $Y$ and
thus we block this junction.

3\. $Z\rightarrow Y\leftarrow C$. This is called “collider” which works in an
exactly opposite way from the above chain and confounding junctions. Once we
know what the value of $Y$ is, $Z$ and $C$ are correlated. However, if we do
not know what $Y$ is or do not intervene it, $Z$ and $C$ are independent and
this junction is naturally blocked.

To sum up, if we want to block a path between two variables, we should
intervene the middle variables in the chain and confounding junctions and
should not intervene in the collider junction. To block a long path, we only
need to block a junction of it, , for $X\leftarrow C\rightarrow Y\leftarrow Z$
in Figure 8(b), we can block $X\leftarrow C\rightarrow Y$ by intervening $C$
or block $C\rightarrow Y\leftarrow Z$ by not intervening $Y$.

###  6.3 The Backdoor Adjustment

The backdoor adjustment is the simplest formula to eliminate the spurious
correlation by approximating the “physical intervention”. Formally, it
calculates the average causal effect of one variable on another at each
stratum of the confounder. For example, in Figure 8(a), we can calculate the
causal effect of $X$ on $Y$ as $P(Y|do(X))$:

$P(Y|do(X))=\sum\nolimits_{c}{P(Y|X,C=c)P(C=c)},$ (9)

where $do(\cdot)$ signifies that we are dealing with an active intervention
rather than a passive observation. The role of Eq. (9) is to guarantee that in
each stratum $c$, $X$ is not affected by $C$ and thus the causal effect can be
estimated stratum by stratum from the data.

###  6.4 The Front-door Adjustment

From Eq. (9), we find that to use the backdoor adjustment, we need to know the
details of the confounder for splitting it into various strata. However, in
our case, we have no idea about what constructs the hidden confounders in the
dataset, thus we are unable to deploy the backdoor adjustment. Fortunately,
the front-door adjustment [43] does not require any knowledge on the
confounder and can also calculate the causal effect between $X$ and $Y$ in a
front-door SCM as in Figure 8(b).

In Section 3.1 of the submitted manuscript, we have shown the derivation of
the front-door adjustment from the attention mechanism perspective. Here we
demonstrate a more formally derivation. The front-door adjustment calculates
$P(Y|do(X))$ in the front-door $X\rightarrow Z\rightarrow Y$ by chaining
together two partially causal effects $P(Z|do(X))$ and $P(Y|do(Z))$:

$P(Y|do(X))=\sum\nolimits_{z}P(Z=z|do(X))P(Y|do(Z=z)).$ (10)

To calculate $P(Z=z|do(X))$, we should block the backdoor path $X\leftarrow
C\rightarrow Y\leftarrow Z$ between $X$ and $Z$. As we discussed in Section
6.2 that a collider junction is naturally blocked and here $C\rightarrow
Y\leftarrow Z$ is a collider, thus this path is already blocked and we have:

$P(Z=z|do(X))=P(Z=z|X).$ (11)

For $P(Y|do(Z))$, we need to block the backdoor path $Z\leftarrow X\leftarrow
C\rightarrow Y$ between $Z$ and $Y$. Since we do not know the details about
the confounder $C$, we can not use Eq. (9) to deconfound $C$. Thus we have to
block this path by intervening $X$:

$P(Y|do(Z=z))=\sum\nolimits_{x}{P(Y|Z=z,X=x)P(X=x)}.$ (12)

At last, by bringing Eq. (11) and (12) into Eq. (10), we have:

$\displaystyle P(Y|do(X))$ (13) $\displaystyle=$
$\displaystyle\sum\nolimits_{z}P(Z=z|X)\sum\nolimits_{x}P(X=x)[P(Y|Z=z,X=x)],$

which is the front-door adjustment given in Eq. (3) of the submitted
manuscript.

##  7 Formula Derivations

Here we show how to use Normalized Weighted Geometric Mean (NWGM)
approximation [71, 58] to absorb the sampling into the network for deriving
Eq. (5) in the submitted manuscript. Before introducing NWGM, we first revisit
the calculation of a function $y(x)$’s expectation according to the
distribution $P(x)$:

$\mathbb{E}_{x}[y(x)]=\sum\nolimits_{x}{y(x)P(x)},$ (14)

which is the weighted arithmetic mean of $y(x)$ with $P(x)$ as the weights.

Correspondingly, the weighted geometric mean (WGM) of $y(x)$ with $P(x)$ as
the weights is:

$\text{WGM}(y(x))=\prod\nolimits_{x}y(x)^{P(x)},$ (15)

where the weights $P(x)$ are put into the exponential terms. If $y(x)$ is an
exponential function that $y(x)=\text{exp}[g(x)]$, we have:

$\displaystyle\text{WGM}(y(x))=\prod\nolimits_{x}y(x)^{P(x)}$ (16)
$\displaystyle=\prod\nolimits_{x}\text{exp}[g(x)]^{P(x)}=\prod\nolimits_{x}\text{exp}[g(x)P(x)]$
$\displaystyle=\text{exp}[\sum\nolimits_{x}g(x)P(x)]=\text{exp}\\{\mathbb{E}_{\bm{x}}[g(x)]\\},$

where the expectation $\mathbb{E}_{x}$ is absorbed into the exponential term.
Based on this observation, researchers approximate the expectation of a
function as the WGM of this function in the deep network whose last layer is a
Softmax layer [71, 58]:

$\mathbb{E}_{x}[y(x)]\approx\text{WGM}(y(x))=\text{exp}\\{\mathbb{E}_{x}[g(x)]\\},$
(17)

where $y(x)=\text{exp}[g(x)]$.

In our case, we treat $P(Y|X,Z)$ (Eq. (3) of the submitted manuscript) as a
predictive function and parameterize it by a network with a Softmax layer as
the last layer:

$\displaystyle P(Y|X,Z)=\text{Softmax}[g(X,Z)]\propto\exp[g(X,Z)].$ (18)

Following Eq. (3) of the manuscript and Eq. (17), we have:

$\displaystyle P(Y|do(X))$ (19) $\displaystyle=$
$\displaystyle\sum\nolimits_{z}P(Z=z|X)\sum\nolimits_{x}P(X=x)[P(Y|Z=z,X=x)]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{[Z|X]}\mathbb{E}_{[X]}[P(Y|Z,X)]\approx\text{WGM}(P(Y|Z,X))$
$\displaystyle\approx$
$\displaystyle\exp\\{[g(\mathbb{E}_{[Z|X]}[Z],\mathbb{E}_{[X]}[X])]\\}.$

Note that, as in Eq. (18), $P(Y|Z,X)$ is only proportional to $\exp[g(Z,X)]$
instead of strictly equalling to, we only have
$\text{WGM}(P(Y|Z,X))\approx\exp\\{[g(\mathbb{E}_{[Z|X]}[Z],\mathbb{E}_{[X]}[X])]\\}$
in Eq. (LABEL:equ:supp_nwgm5) instead of equalling to. Furthermore, to
guarantee the sum of $P(Y|do(X))$ to be 1, we use a Softmax layer to normalize
these exponential units:

$\displaystyle
P(Y|do(X))\approx\text{Softmax}(g(\mathbb{E}_{[Z|X]}[Z],\mathbb{E}_{[X]}[X])),$
(20)

where the first part $\mathbb{E}_{[Z|X]}[Z]$ is In-Sample Sampling (IS-
Sampling) and the second part $\mathbb{E}_{[X]}[X]$ is CS-Sample Sampling (CS-
Sampling). Since the Softmax layer normalizes these exponential terms, this is
called the normalized weighted geometric mean (NWGM) approximation.

In a network, the variables $X$ and $Z$ are represented by the embedding
vectors and thus we use $\bm{x}$ and $\bm{z}$ to denote them. Following the
convention in attention research where the attended vectors are usually
represented in the matrix form, we also pack the estimated IS-Sampling and CS-
Sampling vectors to $\hat{\bm{X}}$, $\hat{\bm{Z}}$. In this way, we have:

$\displaystyle P(Y|do(X))\approx\text{Softmax}[g(\hat{\bm{Z}},\hat{\bm{X}})],$
(21)

which is given in Eq. (5) of the submitted manuscript.

To estimate $\hat{\bm{Z}}$, researchers usually calculate a query set from
$X$: $\bm{Q}_{I}=h(X)$ and use it in the Q-K-V operation. Similarly, to
estimate $\hat{\bm{X}}$, we can also calculate a query set as:
$\bm{Q}_{C}=f(X)$ and use it in the Q-K-V operation. In this way, we have Eq.
(5) in the submitted manuscript:

$\displaystyle P(Y|do(X))\approx{}$
$\displaystyle\text{Softmax}[g(\hat{\bm{Z}},\hat{\bm{X}})],$ (22)
$\displaystyle\textbf{IS-Sampling:}\quad\hat{\bm{Z}}=$
$\displaystyle\sum\nolimits_{z}P(Z=z|h(X))\bm{z},$ $\displaystyle\textbf{CS-
Sampling:}\quad\hat{\bm{X}}=$
$\displaystyle\sum\nolimits_{x}P(X=x|f(X))\bm{x}.$

Note that although $P(X)$ in CS-Sampling does not condition on any variable,
we still require a query in Q-K-V operation, since without a query, the
estimated result will degrade into a fixed single vector for each different
input $X$: $\hat{\bm{x}}=\sum\nolimits_{x}P(x)\bm{x}$, where $P(x)$ is the
prior probability. We can also treat it as the strategy to increase the
representation power of the whole model.

##  8 More Results

###  8.1 Online Captioning Test

We report the performances of the MS COCO online split in Table 7. It can be
found that our single Transformer+CATT can achieve higher performances than
the other state-of-the-art models on this split.

Table 7: The performances of various single models on the online MS-COCO test
server.

Model | B@4 | M | R-L | C-D
---|---|---|---|---
Metric | c5 | c40 | c5 | c40 | c5 | c40 | c5 | c40
BUTD [3] | $36.9$ | $68.5$ | $27.6$ | $36.7$ | $57.1$ | $72.4$ | $117.9$ | $120.5$
CAVP [34] | $37.9$ | $69.0$ | $28.1$ | $37.0$ | $58.2$ | $73.1$ | $121.6$ | $123.8$
RFNet [23] | $38.0$ | $69.2$ | $28.2$ | $37.2$ | $58.2$ | $73.1$ | $122.9$ | $125.1$
SGAE [73] | $37.8$ | $68.7$ | $28.1$ | $37.0$ | $58.2$ | $73.1$ | $122.7$ | $125.5$
CNM [74] | $37.9$ | $68.4$ | $28.1$ | $36.9$ | $58.3$ | $72.9$ | $123.0$ | $125.3$
AoANet† [21] | $37.3$ | $68.1$ | $28.3$ | $37.2$ | $57.9$ | $72.8$ | $124.0$ | $126.2$
Transformer | $37.9$ | $69.2$ | $28.7$ | $37.7$ | $58.3$ | $73.3$ | $124.1$ | $126.7$
Transformer+CATT | $\bm{38.8}$ | $\bm{70.6}$ | $\bm{28.9}$ | $\bm{38.2}$ | $\bm{58.7}$ | $\bm{73.9}$ | $\bm{126.3}$ | $\bm{128.8}$

###  8.2 More Qualitative Examples

Figure 9 shows more qualitative examples where our CATT helps different models
confront the dataset biases. The first two rows show six examples of image
captioning and the last two rows show the examples of VQA. For example, in the
left example of the first row, after incorporating the CATT module, BUTD [3]
generates correctly gender of the person without using the spurious
correlation between “woman” with “kitchen” in the dataset.

Figure 9: More examples demonstrate that our CATT helps various models
confront dataset biases. Red and blue index the incorrect and correct
generated captions and answers, respectively.

##  9 Implementation Details

BUTD + CATT. We deployed this architecture for addressing IC and VQA. In the
original BUTD architecture, they only used one attention module and thus we
also used one causal attention module as in Figure 4. In this architecture, we
set IS-ATT the same as the attention module in BUTD where the probability in
Eq. (6) is calculated as:

$\displaystyle a_{n}=\bm{w}^{T}(\bm{W}_{k}\bm{k}_{n}+\bm{W}_{q}\bm{q}),$ (23)
$\displaystyle\bm{\alpha}=\text{Softmax}(\\{a_{1},...,a_{N}\\}),$

where $\bm{w}$ is a trainable vector and $\bm{W}_{k}$, $\bm{W}_{q}$ are two
trainable matrices. $\bm{V}_{I}$, $\bm{K}_{I}$ were both set to the RoI
feature set of the current image and $\bm{q}_{I}$ was the embedding of the
sentence context, , the partially generated caption or the question for IC or
VQA, respectively. CS-ATT was set to Eq. (7), $\bm{q}_{C}$ was the same as in
IS-ATT and $\bm{V}_{C}$, $\bm{K}_{C}$ were both set to the visual global
dictionary. This dictionary was initialized by applying K-means over all the
RoI features in the training set to get 1000 cluster centres and was updated
during the end-to-end training. The RoI object features were extracted by a
Faster-RCNN [52] pre-trained on VG as in [3]. The hidden size of the LSTM
layers was set to 1024.

For the IC model, the cross-entropy loss and the self-critical reward [53]
were used to train it 35 and 65 epochs, respectively. We used the Adam
optimizer [26] and initialized the learning rate as $5e^{-4}$ and decayed it
by 0.8 every 5 epochs. The batch size was set to 100. For the VQA model, we
followed [63, 3] to use the binary cross-entropy loss and applied the AdaDelta
optimizer [82], which does not require to fix the learning rate, to train it
30 epochs. The batch size was set to 512.

Transformer + CATT. We deployed the architecture in Figure 5 for solving IC
and VQA. In this architecture, the Q-K-V operations of all IS-ATT and CS-ATT
were implemented by 8-head scaled dot product [65]:

Input: $\displaystyle\bm{Q},\bm{K},\bm{V}$ (24) Prob:
$\displaystyle\bm{A}_{i}=\text{Softmax}(\frac{\bm{Q}\bm{W}_{i}^{Q}(\bm{K}\bm{W}_{i}^{K})^{T}}{\sqrt{d}})$
$\displaystyle\textbf{Single-Head}:$
$\displaystyle\bm{H}_{i}=\bm{A}_{i}\bm{V}\bm{W}_{i}^{V},$ Ouput:
$\displaystyle\hat{\bm{V}}=\text{Embed}([\bm{H}_{1},...,\bm{H}_{8}]\bm{W}^{H}),$

where $\bm{W}_{i}^{*}$ and $\bm{W}^{H}$ are all trainable matrices;
$\bm{A}_{i}$ is the soft attention matrix for the $i$-th head; $[\cdot]$
denotes the concatenation operation, and Embed$(\cdot)$ means the feed-forward
network and the residual operation as in [65]. We shared the parameters
between IS-ATT and CS-ATT in each CATT to keep the outputs staying in the same
feature space. Then compared with the original Transformer, the increments of
the trainable parameters only come from the global image and word embedding
dictionaries, which were initialized by applying K-means over the RoI and word
embeddings of the training set. We set the sizes of both dictionaries to 500
and the hidden size of all the attention modules to 512. The RoI object
features were the same as in BUTD+CATT.

For IC, the training included two processes: we first used the cross-entropy
loss and then the self-critical reward to train the captioner 15 and 35
epochs, respectively. The learning rates of two processes were initialized as
$5e^{-4}$ and $5e^{-5}$ and both of them decayed by 0.8 every 5 epochs. The
Adam optimizer was used and the batch size was set to 10. For VQA, we applied
the binary cross-entropy loss and the Adam optimizer to train it 13 epochs. We
followed [79] to set the learning rate to $\textbf{min}(2.5te^{-5},1e^{-4})$,
where $t$ is the training epoch and after 10 epochs, the learning rate decayed
by 0.2 every 2 epochs. The batch size was set to 64.

## References

  * [1] Ehsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi, and Anton van den Hengel.  Counterfactual vision and language learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10044–10054, 2020.
  * [2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In European Conference on Computer Vision, pages 382–398. Springer, 2016.
  * [3] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In CVPR, 2018.
  * [4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [6] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * [7] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  arXiv preprint arXiv:1806.01261, 2018.
  * [8] Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome.  Mutan: Multimodal tucker fusion for visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2612–2620, 2017.
  * [9] Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal.  A meta-transfer objective for learning to disentangle causal mechanisms.  arXiv preprint arXiv:1901.10912, 2019.
  * [10] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In Advances in neural information processing systems, pages 4349–4357, 2016.
  * [11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  arXiv preprint arXiv:2005.12872, 2020.
  * [12] Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In Proceedings of the 37th International Conference on Machine Learning, volume 1, 2020.
  * [13] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [14] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In European Conference on Computer Vision, pages 104–120. Springer, 2020.
  * [15] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-memory transformer for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578–10587, 2020.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [17] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li.  Dynamic fusion with intra-and inter-modality attention flow for visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6639–6648, 2019.
  * [18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
  * [19] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach.  Women also snowboard: Overcoming bias in captioning models.  In European Conference on Computer Vision, pages 793–811. Springer, 2018.
  * [20] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang.  Distilling causal effect of data in class-incremental learning, 2021.
  * [21] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.  Attention on attention for image captioning.  In International Conference on Computer Vision, 2019.
  * [22] Drew A Hudson and Christopher D Manning.  Gqa: a new dataset for compositional question answering over real-world images.  arXiv preprint arXiv:1902.09506, 3(8), 2019.
  * [23] Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang.  Recurrent fusion network for image captioning.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 499–515, 2018.
  * [24] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  * [25] Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang.  Hadamard product for low-rank bilinear pooling.  arXiv preprint arXiv:1610.04325, 2016.
  * [26] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [27] Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath.  Causalgan: Learning causal implicit generative models with adversarial training.  arXiv preprint arXiv:1709.02023, 2017.
  * [28] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123(1):32–73, 2017.
  * [29] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov.  Measuring bias in contextualized word representations.  arXiv preprint arXiv:1906.07337, 2019.
  * [30] Chenliang Li, Ming Yan, Haiyang Xu, Fuli Luo, Wei Wang, Bin Bi, and Songfang Huang.  Sem{vlp}: Vision-language pre-training by aligning semantics at multiple levels, 2021.
  * [31] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [32] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [33] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  Text Summarization Branches Out, 2004.
  * [34] Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, and Feng Wu.  Context-aware visual policy network for sequence-level image captioning.  In 2018 ACM Multimedia Conference on Multimedia Conference, pages 1416–1424. ACM, 2018.
  * [35] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Advances in Neural Information Processing Systems, pages 13–23, 2019.
  * [36] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437–10446, 2020.
  * [37] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Hierarchical question-image co-attention for visual question answering.  In Advances in neural information processing systems, pages 289–297, 2016.
  * [38] Ruotian Luo.  An image captioning codebase in pytorch, 2017.
  * [39] Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich.  Discriminability objective for training descriptive captions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6964–6974, 2018.
  * [40] Moin Nadeem, Anna Bethke, and Siva Reddy.  Stereoset: Measuring stereotypical bias in pretrained language models.  arXiv preprint arXiv:2004.09456, 2020.
  * [41] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen.  Counterfactual vqa: A cause-effect look at language bias.  arXiv preprint arXiv:2006.04315, 2020.
  * [42] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
  * [43] Judea Pearl.  Causal diagrams for empirical research.  Biometrika, 82(4):669–688, 1995.
  * [44] Judea Pearl.  Causality: models, reasoning and inference, volume 29.  Springer, 2000.
  * [45] Judea Pearl and Elias Bareinboim.  External validity: From do-calculus to transportability across populations.  Statistical Science, pages 579–595, 2014.
  * [46] Judea Pearl, Madelyn Glymour, and Nicholas P Jewell.  Causal inference in statistics: A primer.  John Wiley & Sons, 2016.
  * [47] Judea Pearl and Dana Mackenzie.  The Book of Why.  Basic Books, New York, 2018.
  * [48] Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang.  Two causal principles for improving visual dialog.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10860–10869, 2020.
  * [49] Yu Qin, Jiajun Du, Yonghua Zhang, and Hongtao Lu.  Look back and predict forward in image captioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8367–8375, 2019.
  * [50] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training, 2018.
  * [51] William J Reed.  The pareto, zipf and other power laws.  Economics letters, 74(1):15–19, 2001.
  * [52] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [53] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024, 2017.
  * [54] Ronald A Rensink.  The dynamic representation of scenes.  Visual cognition, 7(1-3):17–42, 2000.
  * [55] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  arXiv preprint arXiv:1809.02156, 2018.
  * [56] Donald B Rubin.  Causal inference using potential outcomes: Design, modeling, decisions.  Journal of the American Statistical Association, 100(469):322–331, 2005.
  * [57] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [58] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [59] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  arXiv preprint arXiv:1811.00491, 2018.
  * [60] Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer.  Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.  In International Conference on Machine Learning, pages 6056–6065. PMLR, 2019.
  * [61] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [62] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3716–3725, 2020.
  * [63] Damien Teney, Peter Anderson, Xiaodong He, and Anton Van Den Hengel.  Tips and tricks for visual question answering: Learnings from the 2017 challenge.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4223–4232, 2018.
  * [64] Roger BH Tootell, Nouchine Hadjikhani, E Kevin Hall, Sean Marrett, Wim Vanduffel, J Thomas Vaughan, and Anders M Dale.  The retinotopy of visual spatial attention.  Neuron, 21(6):1409–1422, 1998.
  * [65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [66] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [67] Victor Veitch, Dhanya Sridhar, and David Blei.  Adapting text embeddings for causal inference.  In Conference on Uncertainty in Artificial Intelligence, pages 919–928. PMLR, 2020.
  * [68] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015.
  * [69] Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun.  Visual commonsense r-cnn.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10760–10770, 2020.
  * [70] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [71] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057, 2015.
  * [72] Xiaofeng Yang, Guosheng Lin, Fengmao Lv, and Fayao Liu.  Trrnet: Tiered relation reasoning for compositional visual question answering.  2020\.
  * [73] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai.  Auto-encoding scene graphs for image captioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10685–10694, 2019.
  * [74] Xu Yang, Hanwang Zhang, and Jianfei Cai.  Learning to collocate neural modules for image captioning.  arXiv preprint arXiv:1904.08608, 2019.
  * [75] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola.  Stacked attention networks for image question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21–29, 2016.
  * [76] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Exploring visual relationship for image captioning.  In Computer Vision–ECCV 2018, pages 711–727. Springer, 2018.
  * [77] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo.  Image captioning with semantic attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659, 2016.
  * [78] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-vil: Knowledge enhanced vision-language representations through scene graph.  arXiv preprint arXiv:2006.16934, 2020.
  * [79] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian.  Deep modular co-attention networks for visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6281–6290, 2019.
  * [80] Zhongqi Yue, Tan Wang, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua.  Counterfactual zero-shot and open-set visual recognition.  In CVPR, 2021.
  * [81] Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua.  Interventional few-shot learning.  arXiv preprint arXiv:2009.13000, 2020.
  * [82] Matthew D Zeiler.  Adadelta: an adaptive learning rate method.  arXiv preprint arXiv:1212.5701, 2012.
  * [83] Dong Zhang, Hanwang Zhang, Jinhui Tang, Xiansheng Hua, and Qianru Sun.  Causal intervention for weakly-supervised semantic segmentation.  arXiv preprint arXiv:2009.12547, 2020.
  * [84] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett.  Learning to count objects in natural images for visual question answering.  arXiv preprint arXiv:1802.05766, 2018.
  * [85] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  arXiv preprint arXiv:1909.11059, 2019.