11institutetext: 1Leibniz University of Hannover, Germany 2Freie Universitat
Berlin, Germany
3Moffitt Cancer Center, USA 4Coventry University, United Kingdom
5Dalhousie University, Canada 6Technion – Israel Institute of Technology,
Israel
7Durham University, United Kingdom 8Carnegie Mellon University, USA
11email: {xuejiao.tang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Attention Mechanism based Cognition-level Scene Understanding

Xuejiao Tang 11 Tai Le Quy 11 Eirini Ntoutsi 22 Kea Turner 33 Vasile Palade 44
Israat Haque 55 Peng Xu 66 Chris Brown and Wenbin Zhang 77881122334455667788

###### Abstract

Given a question-image input, the Visual Commonsense Reasoning (VCR) model can
predict an answer with the corresponding rationale, which requires inference
ability from the real world. The VCR task, which calls for exploiting the
multi-source information as well as learning different levels of understanding
and extensive commonsense knowledge, is a cognition-level scene understanding
task. The VCR task has aroused researchers’ interest due to its wide range of
applications, including visual question answering, automated vehicle systems,
and clinical decision support. Previous approaches to solving the VCR task
generally rely on pre-training or exploiting memory with long dependency
relationship encoded models. However, these approaches suffer from a lack of
generalizability and losing information in long sequences. In this paper, we
propose a parallel attention-based cognitive VCR network PAVCR, which fuses
visual-textual information efficiently and encodes semantic information in
parallel to enable the model to capture rich information for cognition-level
inference. Extensive experiments show that the proposed model yields
significant improvements over existing methods on the benchmark VCR dataset.
Moreover, the proposed model provides intuitive interpretation into visual
commonsense reasoning.

##  1 Introduction

Visual understanding is an important research domain with a long history that
attracts extensive models such as Mask RCNN [1], ResNet [2] and UNet [3]. They
have been successfully employed in a variety of visual understanding tasks
such as action recognition, image classification, pose estimation and visual
search [4]. Most of them gain high-level understanding by identifying the
objects in view based on visual input. However, reliable visual scene
understanding requires not only recognition-level but also cognition-level
visual understanding, and seamless integration of them. More specifically, it
is desirable to identify the objects of interest to infer their actions,
intents and mental states with an aim of having a comprehensive and reliable
understanding of the visual input. While this is a natural task for humans,
existing visual understanding systems suffer from a lack of ability for
higher-order cognition inference [5].

To improve the cognition-level visual understanding, recent research in visual
understanding has shifted inference from recognition-level to cognition-level
which contains more complex relationship inferences. This directly leads to
four major directions on cognition-level visual understanding research: 1)
image generation [6], which aims at generating images from given text
description; 2) image caption [7], which focuses on generating text
description from given images; 3) visual question answering, which aims at
predicting correct answers for given images and questions; 4) visual
commonsense reasoning (VCR) [5], which additionally provides rational
explanations along with question answering and has gained considerable
attention [8].

The VCR task is seen as a challenging mountain to climb. To solve the
cognition-level visual task, the model need to learn human’s inference
ability. This might be easy for human as we humans have a reserve of knowledge
and excellent reasoning abilities. However, it is challenging for up-to-date
AI systems. Recently, a large number of researches (see, e.g., [5, 8, 9, 10,
11]) has been proposed to solve the challenging task. These research on VCR
typically necessitates pre-training on large scale data prior to performing
VCR tasks. They usually fit well towards the properties that the pre-training
data possessed but their generalization on other tasks are not guaranteed
[12]. To remove the necessity of pre-training, another line of research
focuses on directly learning the architecture of a system to find
straightforward solutions for VCR [10]. However, these methods suffer
commonsense information loss where the commonsense reasoning ability of the
model is limited. Also, multimodel fusion to fuse visual and textual
information is a daunting task.

To address the aforementioned challenges, we propose a parallel structure-
based model PAVCR, which encodes the commonsense between sentences with less
information lost. We first design a multimodal fusion layer to fuse visual and
textual information. Then, we introduce a commonsense encoder to enhance the
inference ability of the proposed model. Then, we introduce a commonsense
encoder to enhance the inference ability of the proposed model. The novelty of
this research comes from five aspects:

  * -

We theoretically analyze the inadequacy of previous work [13] and introduce a
new effective model to reduce the sequential computation. In [13], the
computational complexity of related signals from two positions grows with the
distance between the positions, which results in difficulties in learning
dependencies among positions for the sequential task. In the newly proposed
multimodal feature fusion layer and commonsense encoder submodules, we
designed a parallel attention structure to limit the number of operations.

  * -

A newly proposed model for VCR task, which represents cognition-level scene
understanding.

  * -

A new multimodal fusion layer that fuses visual and textual information.

  * -

A new commonsense encoder layer with a parallel structure attention mechanism
along with memory cell that avoid information loss while storying the
extracted knowledge extracted commonsense between queries and responses.

  * -

Extensive experiments comparing with poplar methods for VCR tasks and ablation
studies.

This work is an extension of our previous work [14]. The major changes
include: i) a newly proposed multimodal fusion layer for visual-textual
information fusion, ii) a parallel structure based commonsense encoder, which
enable the model capture more information without long dependency, along with
a memory cell to accumulate commonsense storage, iii) case study to discuss
the superior ability of the propose model, iv) ablation study to validate the
proposed multimodal fusion layer and commonsense encoder layer.

The rest of this paper is organized as follows. In section 2 we review related
work on QA (and specifically on VCR). Section 3 briefly covers notation. In
section 4 we detail how the proposed model works to handle VCR task. In
section 5 we apply our model to the VCR dataset and conduct experiments. In
section 6 we show the case study to compare the prediction results from newly
proposed model and the base model. In section 7 we display the quantitative
results predicted by PAVCR. Finally, in section 8 we conclude our paper.

##  2 Related Work

From individual object level scene understanding [1] which aims at object
instance segmentation and image recognition, to visual relationship detection
[15] which captures the relationship between any two objects in image or
videos, state-of-the-art visual understanding models have achieved remarkable
progress [16]. However, that is far from satisfactory for visual understanding
as an ideal visual system necessitates the ability to understand the deep-
level meaning behind a scene. Recent research on visual understanding has
therefore shifted inference from recognition-level to cognition-level which
contains more complex relationship inferences. Rowan et al. [5] further
formulated Visual Commonsense Reasoning as the VCR task, which is an important
step towards reliable visual understanding, and benchmarked the VCR dataset.
Specifically, the VCR dataset is sampled from a large sample of movie clips in
which most of the scenes refer to logic inferences. For example, “Why isn’t
Tom sitting next to David?”, which requires high-order inference ability about
the scene to select the correct answer from available choices. Motivated
studies generally fall into one of the following two categories based on the
necessity of pre-training dataset.

The first line of research, pre-training approaches, trains the model on a
large-scale dataset then fine-tunes the model for downstream tasks. The recent
works include ERNIE-ViL-large [8] and UNITER-large [12]. While the former
learns semantic relationship understanding for scene graph prediction, the
latter is pre-trained to learn joint image-text representations. However, the
generalizability of these models relies heavily on the pre-training dataset
and therefore is not guaranteed.

Another research line focus on studying the model structure to find a
straightforward solution for VCR task. It focuses on encoding the relationship
between sentences using sequence-to-sequence based encoding methods. These
methods infer rationales by encoding the long dependency relationship between
sentences (see, e.g., R2C [5] and TAB-VCR [10], DMVCR [13]). However, these
models face difficulty with reasoning information lost based on long
dependency structure, and it is hard for them to infer reason based on
commonsense about the world. More recently, CAN [14] proposes a co-attention
network to ease model training and enhance the capability of capturing
relationship between sentences and semantic information from surrounding
words, which has already gained a remarkable improvement in VCR task. Our work
is resemble this method which is independent of large-scale pre-training
dataset. Two distinctions in our proposed work are: i) a cross attention based
multimodal unit is designed to fuse visual information from images and textual
information from language sentences, ii) a parallel structure co-attention
network with memory cell for storing commonsense rather than long dependency
structure network to enhance the capability of capturing semantic information
in case of long sentences.

Figure 1: A VCR running example.

##  3 Notations and Problem Formulation

The VCR dataset consists of millions of labeled subsets. Each subset is
composed of an image with one to three associated questions. Each question is
then associated with four candidate answers and four candidate rationales. The
overarching task is formulated as three subtasks: (1) predicting the correct
answer for a given question and image ($Q\rightarrow A$); (2) predicting the
correct rationale for a given question, image, and correct answer
($QA\rightarrow R$); and (3) predicting the correct answer and rationale for a
given image and question ($Q\rightarrow AR$). Additionally, we defined two
language inputs - query $q\\{q_{1},q_{2},\cdots,q_{n}\\}$ and response
$r\\{r_{1},r_{2},\cdots,r_{n}\\}$, as reflected in Figure 2. In the
$Q\rightarrow A$ subtask, query $q$ is the question and response $r$ is the
answers. In the $QA\rightarrow R$ subtask, query $q$ becomes the question
together with correct answer, while rationales constitute the response $r$.
For example, in Figure 1, the $Q\rightarrow A$ subtask in Question 1 predicts
correct answer choice for given image and question. Here, $q$ is the given
question (”Are [0,1] happy to be here?”) and $r$ is the given answer choices
(”A: Yes, they will $\cdots$”, ”B: No, neither of them is happy,$\cdots$”, ”C:
No, [0,1] took the stairs $\cdots$.”, ”D: They both $\cdots$”). Compared to
$Q\rightarrow A$ subtask, the $QA\rightarrow R$ subtask predicts rationale for
given image, question and correct answer. Here, $q$ is the question (”Are
[0,1] happy to be here?”) along with the correct answer (”B: No, neither of
them is happy, and they want to go home.”), $r$ is the rationale choices (”A:
[0] looks distressed $\cdots$”, ”B: [1] is in an argument with [0] $\cdots$”
”C: Both their expression $\cdots$”, ”D: They both $\cdots$”)

##  4 Proposed Framework

Figure 2 illustrates the steps of the learning process by the proposed
framework. It consists of four layers: a feature representation layer, a
multimodal fusion layer, a commonsense encoder layer, and a prediction layer.
The first layer captures language and image features and converts them into
dense representations. The represented features are then fed into the
multimodal fusion layer to generate meaningful contexts of language-image
fused information. Next, the fused features are fed into a commonsense encoder
layer, which consists of a co-attention unit and memory cell to support
commonsense storage. Finally, a prediction layer is designed to predict the
correct answer or rationale.

Figure 2: Proposed framework.

###  4.1 Feature Representation Layer

Extracting informative features from multi-source information plays an
important role in any machine learning application, especially in our context
where the feature itself is one of the learning targets. As shown in Figure 2,
for the image feature extraction, the original image information source is the
image along with its objects, which is given by means of related bounding
boxes serving as a point of reference for objects within the images. The
bounding boxes of given image and objects are then fed into the deep nets to
obtain sufficient information from original image information source.
Concretely, PAVCR extracts image features by a deep network backbone ResNet50
[17] and fine-tunes the final block of the network after RoiAlign. In
addition, the skip connection [2] is adopted to circumvent the gradient
vanishing problem when training the deep nets.

Language embedding. The language embeddings are obtained by transforming raw
input sentences into low-dimensional embeddings. The query represented by
$q\\{q_{1},q_{2},\\\ \cdots,q_{n}\\}$ refers to a question in the question
answering task ($Q\rightarrow A$), and a question paired with correct answer
in the reasoning task ($QA\rightarrow R$). Responses
$r\\{r_{1},r_{2},\cdots,\\\ r_{n}\\}$ refer to answer candidates in the
question answering task ($Q\rightarrow A$), and rationale candidates in the
reasoning task ($QA\rightarrow R$). The embeddings are extracted using an
attention mechanism with parallel structure [18]. Note that the sentences
contain tags related to objects in the image. For example, see Figure 1 and
the question “Are [0,1] happy to be here?” The [0,1] are tags set to identify
objects in the image (i.e., the object features of person 1 and person 2).

Object embedding. The images are filtered from movie clips. To ensure images
with rich information, a filter is set to select images with more than two
objects each [5]. The object features are then extracted with a residual
connected deep network [2]. The output of the deep network is object features
with low-dimensional embeddings $o\\{o_{1},o_{2},\cdots,o_{n}\\}$.

###  4.2 Multimodal Feature Fusion Layer

The multimodal feature fusion layer is illustrated in Figure 3. It aims to
learn visual-textual fused features with semantic discriminative visual
features under the guidance of the textual description without harming their
location ability [19]. After the multimodal layer, the fused features enable
the model to learn the ability in capturing context-level semantics of both
vision and text. It consists of 1) a text branch to supply text features
regarding query and responses with attention information for related object
features from image; 2) a visual branch unit to supply mixed visual-textual
information; 3) a text object fusion unit to enable the capability of the
model in capturing context-level semantics of both visual and textual.

Figure 3: Multimodal Feature Fusion Layer.

Textual Branch Unit. Textual branch unit is designed to supply textual
information while extracting semantic information from around words. To this
end, previous extracted textual features from queries or responses are regards
as value $V_{t}$, query $Q_{t}$ and key $K_{t}$. Recall that the aim of
textual branch unit is to learn textual information while considering around
words. Here, we employ the multi-head attention [20] to obtain attended
information from around words. Formally put,

$X_{1}=V_{t}softmax(\frac{Q_{t}K_{t}}{\sqrt{d}})$ (1)

where $V_{t}$, $Q_{t}$ and $K_{t}$ is the embedded textual features from
queries or responses, $d$ id the dimension of embeddings. It’s illuminated in
Figure 2 as operation ①.

Next, the output $X_{1}$ can supply rich textual information for visual-
textual fusion.

Visual Branch Unit. A visual branch is designed to learn joint visual-textual
features, which fuses textual information into visual features. In details,
the previously extracted object features $O$ are regarded as Query $Q_{v}$,
Keys $K_{v}$ and Values $V_{v}$ for visual branch’s self-attention
computation. To identify the objects and words order, position encodings are
added before weighted attention computations (labeled as “Pos” in Figure 3).
It can be formulated as:

$PE_{pos,2i}=sin(\frac{pos}{\frac{10000^{2i}}{d_{model}}}),PE_{pos,2i+1}=cos(\frac{pos}{\frac{10000^{2i}}{d_{model}}})$
(2)

where $PE$ represents position encoding, $pos$ is the word position in
sentence, $i$ is dimension, $d_{model}$ is the embedding dimension.

Visual sequential order is crucial for semantic information due to it could
result in incorrect meaning in sentence. Therefore, the visual branch unit
first takes the previously extracted object features $O$ as the input of
position encoding. The operation is illuminated in Figure 2 in ④ and ⑥ and can
be formulated as:

$Q_{ve}=K_{ve}=norm(O^{i})+PE(O^{i})$ (3)

Next, the output is fed into $softmax$ operation ② to obtain weighted sum, it
can be formulated as:

$X_{2}=softmax(\frac{Q_{ve}X_{1}}{\sqrt{d}})$ (4)

where $X_{1}$ is textual information which contains weighted attention
information of each words and comes from operation ① in textual unit.

Recall that the aim of the visual branch unit is to learn joint visual-textual
representation. To this end, operation ⑤ concatenates visual-textual
information from the output of operation ③, which can be formulated as:

$Q_{ve}\oplus X_{3}$ (5)

where $\oplus$ represents concatenate operation and $X_{3}$ is the output of
operation ③. It can be formulated as:

$X_{3}=X_{2}X_{1}$ (6)

Next, a multi-head attention in operation ⑦ is deployed to obtain fused visual
representations, which contains weighted attention information from text
features.

Text-Object Fusion Unit. After visual branch unit, the model has learned rich
visual information containing weighted textual information as well as textual
information containing semantic information from around words. To obtain
context-level semantics regarding both textual and visual information, we
leverage another multi-head attention operation to learn text-object
representations.

###  4.3 Commonsense Encoder

After multimodal fusion layer, the visual and textual information has been
fused and output as $q_{o}$ for fused query and object features, $r_{o}$ for
fused response and object features. To further implement VCR task, a
commonsense encoder layer is designed to capture the commonsense between
sentences and use it to enhance inference. As in Figure 4 show, the encoder
contains N co-attention blocks, each block consists of a guided-attention unit
and self-attention unit. Its parallel structure enables the model to capture
the semantic information between sentences in parallel and ease the
information lost problem. Following the N-layer co-attention blocks is a
memory cell to store extracted commonsense.

Figure 4: Co-attention.

Self Attention. Self-attention is designed to capture semantic information
within a sequence. Its structure is shown in Figure 4 as a grey block. The
input consists of Query Q, Keys K, and Values V, which are identical, for the
sake of capturing pairwise relationship in a sequence. In details, pairwise
relationship between samples in a sequence is learned by the multi-head
attention layer. For input sequence
$\widetilde{q}=[\widetilde{q_{1}},\widetilde{q_{2}},...,\widetilde{q_{m}}]$,
the multi-head attention learns the relationship between
$<\widetilde{q_{i}},\widetilde{q_{j}}>$ and outputs attended representations.
Subsequently, the attended representations are transformed by a feed-forward
layer which contains two fully-connected layers with ReLU activation and
dropout. The multihead attention can be formulated as:

$MutiHead=h_{1}\oplus h_{2}\cdots\oplus h_{i}$ (7)

where $h_{i}$ represents an attention head and can be formulated as:

$h_{i}=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (8)

where $d_{k}$ is the dimension size of input embedding K. Q, K, V are input
sequence $\widetilde{q}$

Guided Attention. In comparison to self attention, guided attention focuses on
inter-sentence-wise attention and can be regarded as guided attention learning
weighted information among different sentences. When taking two different
sentences representations $X=[x_{1},x_{2},...,x_{m}]$ and
$Y=[y_{1},y_{2},...,y_{m}]$ as the inputs, $X$ is the query $Q$ while key $K$
and Value $V$ are $Y$, guiding the attention learning for $X$. Specifically,
the multi-head layer in a guided attention unit attends the pairwise
relationship between the two paired input sequences $<x_{i},y_{j}>$ and
outputs the attended representations. A feed-forward layer is then applied to
transform the attended representations. The co-attention network finally
outputs $Z_{q}$ and $Z_{r}$, which are attention information over both images
and texts.

Memory cell. Although the commonsense encoder layer improves long-horizon
sequence modeling by attention mechanism, it would potentially have
difficulties handling continuous inference which requires a knowledge base.
Knowledge base, however, is a crucial requirement for cognition-level
inference. Even human beings, we do inferences based on previous knowledge.

Therefore, we introduce a simple external memory cell to handle this dilemma.
A memory cell is read-writable and learnable. We define the memory cell as
$M$, where $M(i)$ has the same size as $Z_{q}$ or $Z_{r}$. At time step $t$,
the model firstly read memory from $M$ regarding the past knowledge with the
function $f_{read}(M(t-1))$ and then concatenate it with the current embedding
$h_{t}$, which is defined as $f_{write}(M(t))$ This allows the model to
condition current embeddings on the previous embeddings for a consistent
inference. Formally, put:

$f_{read}(M(t-1))=Z_{q/r}^{1}\oplus Z_{q/r}^{2}\cdots\oplus Z_{q/r}^{t-1}$ (9)

where $\oplus$ represents concatenate operation.

$m_{q/r}=f_{write}(M(t))=f_{read}(M(t-1))\oplus Z_{q/r}^{t}$ (10)

where $m_{q/r}$ represents the memory cell output of querries or responses.

###  4.4 Prediction Layer

The prediction layer generates a probability distribution of responses from
the high-dimension context generated in the encoder layer. It consists of an
attention reduction unit and a prediction unit.

Attention Reduction. The commonsense encoder layer includes N layers co-
attention operation. However, some of these are unnecessary for prediction.
Therefore, an attention reduction unit is designed to pick up most significant
information. It can be formulated as:

$\widetilde{Z_{l}}=\sum_{i=1}^{m}\alpha_{l}^{i}z_{l}^{i},~{}\alpha=softmax(MLP(Z_{l}))$
(11)

where $Z_{l}$ is either input query or sequence, $\alpha$ is the learned
attention weights and $i$ is the position in a sequence.

For better gradient flow through the network, PAVCR also fuses the features by
using LayerNorm on the sum of the final attended representations,

$c=LayerNorm(W_{x1}^{T}\widetilde{Z_{q}}+W_{x2}^{T}\widetilde{Z_{r}})$ (12)

where $W_{x1}^{T}$ and $W_{x2}^{T}$ are two trainable linear projection
matrices.

Prediction. The prediction includes a multi-layer perceptron (Dropout(0.3) -
FC (1024) \- ReLU - Dropout(0.3) - FC(1)). VCR task needs to predict the
correct choices from four given choices (answer candidates and reason
candidates). Hence, we treat it as a multi-class task. A popular loss function
for multi-classification task cross-entropy [21] is therefore applied to
complete the prediction. It aims to output probability distributions for four
candidate choices and the choice with the max prediction probability is the
final prediction.

##  5 Experimental Results

In this section, we conduct extensive experiments to demonstrate the
effectiveness of our proposed PAVCR network for solving VCR tasks. We first
introduce the datasets, baseline models, and evaluation metrics of our
experiments. Then we compare our model with baseline models and present an
analysis of the impact of the different strategies. Finally, we present an
intuitive interpretation of the prediction. The experiments were conducted on
a 64-bit machine with a 10-core processor (i9, 3.3GHz), 64GB memory with GTX
1080Ti GPU.

###  5.1 Dataset

The VCR dataset [5] consists of 290k multiple-choice questions, 290k correct
answers, 290k correct rationales, and 110k images. The correct answers and
rationales are labeled in the dataset with $>90\%$ of human agreements. As
shown previously in Figure 1, each set consists of an image, a question, four
available answer choices, and four reasoning choices. The correct answer and
rationale are provided in the dataset as ground truth.

Figure 5: Overview of the types of inference required by questions in VCR.

In addition, the dataset distribution is shown in Figure 5. 38% of the types
of inference is about explanation. 24% is about activity inference. These
types of tasks represent cognition-level inference tasks.

###  5.2 Metric.

The VCR task can be regarded as a multi-classification problem. We use mAp
[21] to evaluate the performance, which is a common metric for evaluating
prediction accuracy in multi-classification areas. The mAP is usually computed
on a dataset.

###  5.3 Approach for comparison.

We compare the proposed PAVCR with recent deep learning-based models for VCR.
Specifically, the following baseline approaches are evaluated:

  * -

RevisitedVQA [22]: While the most recently proposed VCR methods have reasoning
modules, RevisitedVQA deploys logistic regressions and multi-layer perceptrons
(MLP) for reasoning tasks.

  * -

BottomUpTopDown [23]: BottomUpTopDown uses weighted feature information over
language and image to predict answers and reasons.

  * -

MLB [24]: The key operation for MLB is the Hadamard product for attention
mechanism, which enables a low-rank bilinear pooling for the task.

  * -

MUTAN [25]: MUTAN consists of a multimodal fusion module tucker decomposition
and a multimodal low-rank bilinear (MLB). The MLB is regarded as a reasoning
module for inference.

  * -

R2C [5]: R2C represents a general baseline for VCR tasks. It consists of a
fusion module, a contextualization module, and a reasoning module for
cognition-level inference. It encodes the sequence based on the sequence
relationship model LSTM and attention mechanism.

  * -

DMVCR [13]: Storing commonsense between sentences using working dynamic memory
as dictionary. In the inference stage, the dictionary module looks up
information from the dictionary as well as updates the information in the
dictionary.

  * -

CAN [14]: Proposed a parallel co-attention based network to enhance the
capability of capturing information from surrounding words.

###  5.4 Analysis of Experimental Results

Task description. We implement the experiments separately in three steps. We
firstly conducted $Q\rightarrow A$ evaluation, and then $QA\rightarrow R$.
Finally, we join the $Q\rightarrow A$ result and $QA\rightarrow R$ results to
obtain the final $Q\rightarrow AR$ prediction result. The difference between
the implementation of $Q\rightarrow A$ and $QA\rightarrow R$ tasks is the
input query and response. For the $Q\rightarrow A$ task, the query is the
paired question, image, four candidate answers; while the response is the
correct answer. For the $QA\rightarrow R$ task, the query is the paired
question, image, correct answer, and four candidate rationales; while the
response is the correct rationale.

Analysis. We compare our method with several popular visual scene
understanding models based on the mean average precision metric for the three
subtasks: $Q\rightarrow A$, $QA\rightarrow R$, and $Q\rightarrow AR$,
respectively. As the results in Table 1 showing, our approach outperforms in
all of the subtasks: $Q\rightarrow A$, $QA\rightarrow R$, and $Q\rightarrow
AR$. Specifically, our method outperforms MUTAN and MLB by a large margin due
to that MUTAN and MLB are lacking a commonsense reasoning module, which works
for cognition-level inference. Furthermore, it also performs better than
recently proposed DMVCR [13] and CAN [14]. The reason is that the proposed
PAVCR deploys a more efficient multimodal fusion layer and commonsense encoder
layer.

Table 1: Comparison of results between our methods and other popular methods using the VCR Dataset. The best performance of the compared methods is highlighted. Percentage in parenthesis is our relative improvement over the performance of the best baseline method. Models | $Q\rightarrow A$ | $QA\rightarrow R$ | $Q\rightarrow AR$
---|---|---|---
RevisitedVQA [22] | 39.4 | 34.0 | 13.5
BottomUpTopDown [23] | 42.8 | 25.1 | 10.7
MLB [24] | 45.5 | 36.1 | 17
MUTAN [25] | 44.4 | 32.0 | 14.6
R2C [5] | 61.9 | 62.8 | 39.1
DMVCR [13] | 62.4 | 67.5 | 42.3
CAN [14] | 71.1 | 73.8 | 47.7
PAVCR |  c73.1 |  |
(+3%) |  c74.2 |  |
(+0.54%) |  c49.2 |  |
(+3.1%) |  |  |

This is expected as PAVCR incorporates a more effective visual grounding
module in its encoder network to enhance visual-textual fusion. In addition,
to alleviate the lost information when encoding a long dependence structure
for long sentences of other methods, PAVCR further encodes semantic
information in parallel to capture more comprehensive information from
surrounding words, which also leads to superior performance over the others.

Table 2: Comparison of results between our methods and other popular methods using the VCR Dataset. The best performance of the compared methods is highlighted. Percentage in parenthesis is our relative improvement over the performance of the best baseline method. Models | $QA\rightarrow R$ | $Q\rightarrow AR$
---|---|---
Without multimodal module | 40.2 | 38.2
Dictionary based encoder | 63.4 | 69.1
PAVCR | 73.1 | 74.2

###  5.5 Ablation Study

We also perform ablation studies to evaluate the performance of the proposed
multimodal fusion layer and the commonsense encoder layer. As in Table 2
showing, when we take out the multimodal fusion layer, the prediction result
decreases to 40.2% in $QA\rightarrow R$ task and 38.2 in $Q\rightarrow AR$
task. The large margin decrease indicates that the proposed multimodal fusion
layer can help the model improve the capability of capturing visual-textual
information. In addition, we also conduct ablation study to prove the
effectiveness of the proposed commonsense module. If the proposed encoder
layer is replaced by a dictionary-based encoder, the performance will decrease
9.7% in $QA\rightarrow R$ task and 5.1% in $Q\rightarrow AR$. Compared to the
dictionary-based method, the proposed encoder layer captures information from
surrounding words in parallel, which results in less information loss, as it
doesn’t have a long dependency in long sequential prediction tasks.

Figure 6: Case study example 1. The model predicts the correct answer and
rationale.

##  6 Case Study

In this section, the case study is conducted to compare the performance of the
proposed method and analyze the superiority of the model. Figure 6 shows
prediction results from previous work and the newly proposed work. The results
from the previous work are marked in red, while the results from the newly
proposed model are marked in green. The model predicts two questions for the
given image. Both of the two questions regards human activity prediction.

Figure 7: Qualitative example 1. The model predicts the correct answer and
rationale.

Analyze the results displayed in Figure 6, we can come to the result that the
newly proposed model works better than the previous work CAN [14], due to that
CAN [14] predicts the wrong answer but the correct rationale in Question 2 for
the given image. However, compared with the result, our newly propose PAVCR
correctly predicts all of the choices both in answer and rationale, which
proved the superior ability of our newly proposed model. A crucial factor is
that the proposed multimodal feature fusion layer with visual branch and
textual branch is more powerful in learning visual-textual fusion information
due to it considers not only the multimodal fusion but also pay attention to
the sequential order, which is an important factor in semantic meaning.

Figure 8: Qualitative example 2. The model predicts the correct answer and
rationale.

##  7 Qualitative Results

We evaluate qualitative results on the PAVCR model to show the prediction
results visualization. The qualitative examples are provided in Figure 7, 8,
9. The candidate in green represents the correct choice; the candidate with a
green checkmark represents the prediction result by our proposed PAVCR model.
As the qualitative results show, the PAVCR model has a strong ability in both
question answering and reasoning tasks. It proves the inference ability of the
proposed model in cognition-level visual understanding.

Figure 9: Qualitative example 3. The model predicts the correct answer and
rationale.

In Figure 7, PAVCR performs well in both questions answering and answer
reasoning. In detail, the question listed is: “What kind of profession does
[0,1] and [2] practice?”. The predicted answer is D - “They are all lawyers.”
Furthermore, the model offers rationale C - “[0,1] and [2] are all dressed in
suits and holding papers or briefcases, and meet with people to discuss their
cases.” PAVCR correctly infers the rationale based on dress and activity, even
though this task is also difficult for humans. It proves the inference ability
of the proposed model in cognition-level visual understanding.

PAVCR can also identify human beings’ necessary and infer emotion. See for
example the result in Figure 8. Question 1 is: “Does [0] require medical
attention?”. Our model selects the correct answer A along with reason A: “[0]
does require medical attention”; because “Humans are not typically in hospital
beds unless they require medical attention.”

Moreover, it can also predict human activities. For instance in Figure 9, the
question is “What is [1] doing?” The proposed PAVCR predicts the correct
answer B - “[1] is getting chemotherapy.” along with the reason A - “He is in
a hospital with tubes in him.” The results mean that the proposed model can
analyze activities based on the surrounding environment. This is important and
provides the possibility in a real-world application.

##  8 Conclusion

This work has studied the widely applicable visual commonsense reasoning. To
solve the challenging cognition-level visual scene understanding task, we
propose novel attention-based multimodal fusion layer and a commonsense
encoder layer composed of a feature representation layer to capture multiple
features containing language and objects information; a multimodal fusion
layer to fuse features from language and images. We also conducted extensive
experiments including comparison results with popular VCR models, case study
analysis and ablation study on the VCR dataset to demonstrate the
effectiveness of our model and present intuitive interpretation. A future
direction is to extend the proposed framework in conjunction with our previous
works [26, 27] to investigate various perspectives of bias in visual
reasoning.

## Acknowledgement

This work was supported by a NVIDIA GPU Grant.

## References

  * [1] A. O. Vuola, S. U. Akram, and J. Kannala, “Mask-rcnn and u-net ensembled for nuclei segmentation,” in _IEEE ISBI_ , 2019, pp. 208–212.
  * [2] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on CVPR_ , 2016, pp. 770–778.
  * [3] R. L. Barkau, “Unet: One-dimensional unsteady flow through a full network of open channels. user’s manual,” Hydrologic Engineering Center Davis CA, Tech. Rep., 1996.
  * [4] G. Papandreou, T. Zhu _et al._ , “Towards accurate multi-person pose estimation in the wild,” 2017.
  * [5] R. Zellers, Y. Bisk _et al._ , “From recognition to cognition: Visual commonsense reasoning,” in _Proceedings of the IEEE Conference on CVPR_ , 2019.
  * [6] K. Gregor and I. e. a. Danihelka, “Draw: A recurrent neural network for image generation,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 1462–1471.
  * [7] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in _Proceedings of the IEEE conference on CVPR_ , 2015, pp. 3156–3164.
  * [8] F. Yu, J. Tang _et al._ , “Ernie-vil: Knowledge enhanced vision-language representations through scene graph,” 2020.
  * [9] H. Ben-younes, R. Cadène, M. Cord, and N. Thome, “MUTAN: multimodal tucker fusion for visual question answering,” 2017.
  * [10] J. Lin, U. Jain _et al._ , “TAB-VCR: tags and attributes based VCR baselines.”
  * [11] J. Lu, D. Batra _et al._ , “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13–23.
  * [12] Y.-C. Chen, L. Li _et al._ , “Uniter: Learning universal image-text representations,” 2019.
  * [13] X. Tang, X. Huang, W. Zhang, T. B. Child, Q. Hu, Z. Liu, and J. Zhang, “Cognitive visual commonsense reasoning using dynamic working memory,” in _International Conference on Big Data Analytics and Knowledge Discovery_. Springer, 2021, pp. 81–93.
  * [14] X. Tang, W. Zhang, Y. Yu, K. Turner, T. Derr, M. Wang, and E. Ntoutsi, “Interpretable visual understanding with cognitive attention network,” in _International Conference on Artificial Neural Networks_. Springer, 2021, pp. 555–568.
  * [15] X. Yang, K. Tang _et al._ , “Auto-encoding scene graphs for image captioning,” in _Proceedings of the IEEE Conference on CVPR_ , 2019, pp. 10 685–10 694.
  * [16] J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the kinetics dataset,” 2017.
  * [17] Y. You, Z. Zhang _et al._ , “Imagenet training in minutes,” in _Proceedings of the 47th International Conference on Parallel Processing_ , 2018, pp. 1–10.
  * [18] J. Devlin, M.-W. Chang _et al._ , “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2018.
  * [19] Y. Du, Z. Fu, Q. Liu, and Y. Wang, “Visual grounding with transformers,” 2021\.
  * [20] A. Vaswani, N. Shazeer _et al._ , “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [21] P. Henderson and V. Ferrari, “End-to-end training of object class detectors for mean average precision,” in _Asian Conference on Computer Vision_. Springer, 2016, pp. 198–213.
  * [22] A. Jabri, A. Joulin, and L. Van Der Maaten, “Revisiting visual question answering baselines,” in _European conference on computer vision_. Springer, 2016, pp. 727–739.
  * [23] P. Anderson, X. He _et al._ , “Bottom-up and top-down attention for image captioning and visual question answering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.
  * [24] J.-H. Kim, K.-W. On, W. Lim, J. Kim, J.-W. Ha, and B.-T. Zhang, “Hadamard product for low-rank bilinear pooling,” 2016.
  * [25] H. Ben-Younes, R. Cadene, M. Cord, and N. Thome, “Mutan: Multimodal tucker fusion for visual question answering,” in _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * [26] W. Zhang and J. Weiss, “Longitudinal fairness with censorship,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * [27] W. Zhang, J. C. Weiss, S. Zhou, and T. Walsh, “Fairness amidst non-iid graph data: A literature review,” _arXiv preprint arXiv:2202.07170_ , 2022.