# Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning
Based on Visually Grounded Conversations

Kilichbek Haydarov1 Xiaoqian Shen1 Avinash Madasu1 Mahmoud
Salem${}^{1}\,\,\,\,\,\,\,\,\,\,$ Li-Jia Li2 Gamaleldin Elsayed3 Mohamed
Elhoseiny1 King Abdullah University of Science and Technolgy (KAUST) 1
HealthUnity Corporation2 Google DeepMind3
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We introduce Affective Visual Dialog , an emotion explanation and reasoning
task as a testbed for research on understanding the formation of emotions in
visually grounded conversations. The task involves three skills: (1) Dialog-
based Question Answering (2) Dialog-based Emotion Prediction and (3) Affective
emotion explanation generation based on the dialog. Our key contribution is
the collection of a large-scale dataset, dubbed AffectVisDial, consisting of
50K 10-turn visually grounded dialogs as well as concluding emotion
attributions and dialog-informed textual emotion explanations, resulting in a
total of 27,180 working hours. We explain our design decisions in collecting
the dataset and introduce the questioner and answerer tasks that are
associated with the participants in the conversation. We train and demonstrate
solid Affective Visual Dialog baselines adapted from state-of-the-art models.
Remarkably, the responses generated by our models show promising emotional
reasoning abilities in response to visually grounded conversations. Our
project page is available through https://affective-visual-dialog.github.io.

##  1 Introduction

Figure 1: AffectVisDial captures constructed emotion attributions and
explanations from both the Questioner (without image access) and Answerer
(with image access) after 10 turns of questions and answers starting from two
opposing opinions. The Questioner is then shown the image and given the
opportunity to change their initial emotional response, with a corresponding
textual explanation.

Emotions play a vital role in shaping human experiences, impacting our
perception, attention, memory, and decision-making [27, 59]. Sensory
information is one major factor that modulates human emotions, with visual art
being a significant influencer of human emotions for centuries. As AI systems
become increasingly ubiquitous, it is becoming essential to consider the
emotional aspects of human nature to develop systems that can flexibly respond
in a natural way based on perceived emotions, ultimately increasing the social
acceptance of AI and better supporting humans. Stuart Russel [47] critiqued
the normative AI development model warning of potential catastrophes due to
its frequent disregard of compatibility with human values and goals. In this
work, our focus is to take a step towards developing AI systems compatible
with our emotional being. To develop these emotion-aware AI systems, there is
an eminent need for public affective datasets that capture different sensory
modalities that give rise to human emotions.

In recent years, there has been growing attention to understanding the impact
of visual information on the affective experience of viewers and explanation
in language [5, 37, 36, 4]. While existing research has predominantly centered
on investigating the direct influence of visual stimuli on emotions expressed
via language, a notable void exists in the comprehensive examination of the
impact of visually grounded language on affective responses.

Our research aims to bridge this gap by investigating the formation of
emotions in visually grounded conversations and elucidating the way in which
such emotions are conveyed and explained through language. We believe that
analyzing the interplay between visual cues and spoken language in
conversational settings can provide unique insights into how emotions are
expressed, perceived, and understood during interactive human communication.
This type of data can be valuable for training models using Human Feedback
(e.g., Reinforcement Learning with Human Feedback), as it provides rich
interactions that can guide the model’s responses and emotional understanding
through iterative learning from human interactions. Additionally,
incorporating such data into the training process has the potential to enhance
the emotion awareness of current models, enabling them to better comprehend
based on language conversations.

Why dialog setup? The question-answer-based dialog setup was chosen over
providing text descriptions about visual stimuli because this setup offers
unique advantages that enable a more dynamic and interactive exploration of
the visual content. This allows the questioner to obtain specific information,
seek clarification, and delve deeper into particular aspects of the content,
leading to a more comprehensive understanding of the visual stimulus.

We begin this effort by building a large-scale dataset that captures language
and vision modalities and their influence on human emotions. Specifically, we
are interested in language in the form of a dialog about a visual signal that
guides human emotion. The dialogues capture live conversations between two
people and their constructed emotions in response to visual stimuli. Such
dataset can help us to develop a better understanding of how language (and
vision) information is associated with emotion and can help us build better
models and evaluate them. To that end, we design a novel task to establish a
dialogue between two human participants (a _Questioner_ and an _Answerer_) to
reach an emotional judgment. Our task enables models to produce emotions and
explanations to them from two perspective: (1) Questioner relying only on
language in the form of visually grounded conversation, (2) Answerer relying
on vision and language modalities. Our dataset offers an opportunity to
investigate the shift in affective experience in response to direct/indirect
(e.g. image is not visible) access to visual stimuli. We can explore how
emotions are constructed based the conversation about the hidden image and how
they may change once the image is revealed. The answers provided by the
answerer in the visually grounded conversations were found to be highly
informative, offering sufficient context about the hidden image.
Interestingly, in approximately 23% of cases, even after the image was
revealed to the questioner, their initial emotional response remained
unchanged. Furthermore, our dataset enables us to examine the impact of
continued exposure to visual stimuli on the Answerer’s emotional state during
the conversation. For example, Fig. 1 shows a shift in the questioner’s
emotion response from “fear” to “contentment” once they saw the hidden visual
stimuli.

We also introduce a benchmark on top of AffectVisDial with standard splits and
baseline models for the _Affective Visual Dialog_ Task: evaluating (1)
Question Answering ability and (2) Emotion Classification and Explanation for
the Questioner and Answerer. In addition, we explore the emotion awareness
ability of the foundation model. Although gpt-4 [2] has demonstrated
remarkable proficiency in zero-shot question-answering tasks, experimental
results reveal its difficulty in comprehending emotional cues through dialogue
at a human level. This underscores the necessity of our proposed dataset for
advancing applications such as training emotion-aware AI systems and
comprehending conceptual human expressions like art.

In summary, we make the following contributions:

  * •

We collected a large-scale dialog dataset (AffectVisDial) that captures the
conversation of two participants and their emotional reactions, containing
50,000 unique dialogs with emotion judgments and their explanations.

  * •

We design a novel task of Affective Visual Dialog covering three subtasks: 1)
Dialog-based question answering, 2) Dialog-based emotion classification, (3)
and affective explanation where the system should be able to produce emotion
and explanation in language.

  * •

We introduce several baselines and measure their ability to predict emotions
based on language and/or vision, and most importantly explain why.

  * •

Our findings show that emotion can be influenced by the different responses
within the dialogue, demonstrating the importance of capturing the
conversation context based on visuals for emotion prediction. We conclude by
illustrating in the discussion how the introduced models could potentially
guide conversations according to a desired emotion, and edit the visual
stimuli based on the stipulated emotional reasoning.

##  2 Related Work

Vision-Language datasets. Several notable datasets (e.g. [31, 7, 50, 57]) have
been curated to explore the link between visual and linguistic information,
facilitating the investigation of tasks such as image captioning, visual
question answering (VQA), and visual dialog. Prominent among these datasets is
the MSCOCO dataset [31], which contains a vast collection of images, each
paired with human-generated captions, enabling the training and evaluation of
image captioning models. Another significant dataset is VQA [7], where
questions related to images are accompanied by human-generated answers,
fostering research on bridging vision and language understanding.

Visual Dialog datasets (e.g., VisDial [15] and GuessWhat?! [16]) focus on
generating natural language responses to questions about images, given an
image, dialog history consisting of a sequence of question-answer pairs, and a
follow-up question. Compared to other vision and language tasks, this task is
more challenging and requires visual understanding, reasoning, and text
comprehension from ML systems. For diagnostic purposes, synthetic dialog
datasets [49, 26, 3] were introduced to analyze models in the visual
coreference resolution [46, 25] and multi-round visual reasoning tasks [26].
GuessWhat?! [16] is a game-like dataset in which one player selects an object
in an image and the other player asks questions to guess the object’s
identity. Image Chat [51] was introduced for the purpose of making
conversational agents more engaging by assigning different personality roles
to them. Unlike existing works, our AffectVisDial dataset focuses on the
construction of emotion and the explanation of the affective state of agents
based on visually grounded conversations.

Emotion Datasets. Numerous efforts have been made to collect datasets and
analyze human emotions across different modalities, encompassing facial
expression [38], body gestures [32, 23, 45, 21], text [53, 17, 10], music [8,
20, 24, 14] and visual art [5, 37]. However, most of them were limited in size
or concerned with single modalities. Although some recent studies have
explored the link between emotion attributes, language, and visual stimuli [5,
4, 36], they often lack explanatory language based on linguistic cues from
conversations surrounding hidden visual stimuli. While there are dialog
datasets that address human emotions [30, 12], they lack grounding in visual
signals, which sets our work apart. Our dataset aims to bridge this gap and
is, to the best of our knowledge, the first to focus on capturing emotional
responses and affective expressions through dialogues centered around visual
stimuli.

Figure 2: An example from AffectVisDial dataset.

##  3 AffectVisDial Dataset and Analysis

Dataset Collection. We employed a live communication protocol, where two
agents engage in a dialogue. The first agent (Questioner), asks questions
related to a hidden image, which is intentionally kept hidden to mitigate any
visual priming biases in which models operate under the assumption that the
questioner will inquire about the objects depicted in the image [6, 22]. The
objective of the Questioner is to explore the hidden image. On the other hand,
the second agent (Answerer) observes the image and provides answers to the
questions posed by the Questioner. The unique setup in our user interface
design is that to initiate the conversation, we reveal two opposing opinions
(a negative and a positive caption) from the combined ArtEmis v1 and v2
datasets [5, 37] associated with the artwork. Our intuition of triggering the
dialog with the two opposite opinions is to counter the biases of the
Questioner when starting the task and encourage more open-mindedness towards
the emotion triggered by the hidden visual signal (i.e., open possibility that
the hidden visuals may be perceived subjectively positively or negatively).
After 10 question-answer pair exchanges, the Questioner is asked to provide an
emotion class response and explanation for what informed the Questioner’s
decision based on the dialog. Then, the hidden image is revealed to the
Questioner, and both agents are required to indicate their emotional responses
in the presence of visual cues and conversation. This final question after
revealing the image allows explorations regarding the emotion arising from
only dialog vs. those that are also informed by visual stimuli. For the live
chat framework, we use Mephisto tool [54] with our customized front-end
interfaces. Fig. 2 shows an example of collected dialog; more examples and
interfaces are attached in the supplementary.

Visual Stimuli. We use visual artworks from WikiArt [13] to ground
conversations between two agents. The diversity of artworks in terms of time
periods, art genres (landscape, portrait, still life, etc.), and styles
(abstract, baroque, cubism, impressionism, etc.) naturally allows for engaging
and distinct conversations. We decided to use visual art as the basis of our
dataset for three primary reasons. First, artworks are often created with the
intention of eliciting emotional responses in viewers, making them a natural
choice for studying the relationship between visual stimuli and emotions.
Second, most existing datasets that feature individual emotion attributions
and explanations are centered around art (e.g. [5, 37, 36]). Thirdly, the
subjective interpretation of visual stimuli from the Answerer’s perception
adds an element of diversity to the emotional reactions captured in our
dataset. However, it is worth noting that our data collection protocol can be
applied to any type of visual stimuli, which constructs emotion experience.
For instance, dialogs with real images can be found in the supplementary.

Emotion Set: We adopted the methodology used in prior works (e.g. [56, 55, 35,
5, 37, 36]) and utilized the same set of eight Ekman emotion categories: the
four negative emotions (anger, disgust, fear, sadness) are considered
universal and basic, as initially proposed by Ekman [19], the four positive
emotions (amusement, awe, contentment, excitement) are more nuanced variations
of happiness [18].

Data Inclusion and Exclusion are crucial steps in building any dataset of high
quality. In our data inclusion process, we only consider the dialogs that meet
our criteria of quality and relevance. Specifically, we include only those
dialogs that contain the full 10 turns, where both Questioner and Answerer
provide their emotional explanations about the hidden image. Out of 107,912
dialogs, 17,435 dialogues that had less than 10 turns were deemed “incomplete”
and were excluded from the dataset. We also exclude those dialogs that contain
any inappropriate or irrelevant content that does not follow the given
instructions such as avoiding offensive messages, chitchatting, etc. The
remaining 90,477 complete dialogs were manually inspected, and 40,477 were
excluded for noncompliance with said guidelines. To facilitate a more
productive dialogue that delves deeper into the exploration of the hidden
image, we required Questioners to avoid redundant questions that can be easily
answered by referring to the given opinions. By carefully selecting and
filtering the data, we aim to ensure the quality and usefulness of the
AffectVisDial dataset.

Datasets CLEVR Dialog [26] MNIST Dialog [49] VisDial [15] AffectVisDial (Ours)
Type (real-R, synthetic-S) S S R R # Images 85K 50K 123K 50K # Dialogs 425K
150K 123K 50K # Questions 4.25M 1.5M 1.2M 500K # Unique Q 73K 335 380K 193K #
Unique A 29 38 340K 324K Vocab. Size 125 54 7K 21k Mean Q Len 10.6 8.9 5.1
10.92 Mean A Len 1 1 2.9 14.85 Mean Q Explanation Len NA NA NA 27.29 Mean A
Explanation Len NA NA NA 28.95 Emotion labels NA NA NA 50K $\times 2$ Emotion
Explanations NA NA NA 50K $\times 2$

Table 1: Comparison with existing visual dialog datasets.

Dataset Type Nouns Pronouns Adjectives Adpositions Verbs VisDial [15] Q 1.4
0.6 0.3 0.3 1.4 AffectVisDial (Ours) Q 2.6 1.0 0.5 1.4 2.1 1-7[1.5pt/2pt]
VisDial [15] A 0.9 0.2 0.3 0.2 0.5 AffectVisDial (Ours) A 3.6 1.0 1.1 1.9 2.7
AffectVisDial (Ours) Positive Q 2.6 1.1 0.5 1.4 2.0 Negative Q 2.6 0.9 0.5 1.4
2.0 Positive A 3.5 0.9 1.1 1.8 2.6 Negative A 3.4 1.0 1.1 1.8 2.6 Qers’ Exp
w/o image 5.9 2.6 2.3 3.3 5.8 Qers’ Exp w/ image 3.9 2.0 1.61 2.4 4.0 Aers’
Exp 6.3 2.8 2.6 3.4 6.1

Table 2:  Language richness reported as the average Part-of-Speech units per
individual Questions (Q), Answers (A), and Explanations (Exp) from Questioners
(Qers) and Answers (Aers).

###  3.1 Dataset Analysis

In this section, we analyze our AffectVisDial dataset and compare it to
existing visual dialog datasets. In total, our AffectVisDial dataset contains
50,000 dialogs (10 QA pairs) on $50K$ unique artworks, resulting in 500,000 QA
pairs in total.

Comparison to similar datasets. Tab. 1 compares our dataset to existing visual
dialog datasets. CLEVR [26], MNIST [49] contain more dialogs. However, they
are generated synthetically for the purpose of diagnosis of the Visual Dialog
models and do not capture the real natural conversation. Compared to VisDial
[15], our dataset contains emotion labels and corresponding explanations for
50k dialogs, while VisDial does not. Additionally, our dataset features longer
questions and answers, with a vocabulary size three times larger than VisDial
at 21k. The mean question and answer lengths in AffectVisDial are 10.92 and
14.85, respectively, compared to 5.1 and 2.9 in VisDial.

Linguistic analysis. Tab. 2 presents the Part-of-Speech (PoS) analysis for
questions and answers in AffectVisDial and VisDial. Compared to VisDial,
AffectVisDial has a higher frequency of PoS tags in questions and answers,
suggesting that the answers are more detailed and descriptive of the image
attributes and activities. Emotional explanations in AffectVisDial have higher
PoS tags than questions and answers. This is expected since the agents should
be expressive in their explanations in order to convey what constructed their
emotion from the dialog. We also break down the results for dialogs concluded
with a positive emotion (i.e., amusement, awe, excitement, contentment) $vs$
negative emotion (i.e., sad, anger, disgust, fear). We found that there are
not many statistical differences between questions from dialogs with positive
conclusions and questions from dialogues with negative conclusions.

(a)

(b)

Figure 3: a) Distribution of lengths for questions, answers, and explanations
b) percent coverage of overall answers from the training dataset covered top
frequent unique answers, compared to VisDial. Based on the coverage,
AffectVisDial ’s answers are more diverse (i.e., less percentage of our
AffectVisDial is covered by the same set of unique answers).

Figure 4: Distribution of first n-grams for AffectVisDial (a) questions, (b)
answers, (c) explanations. Word ordering starts towards the center and
radiates outwards, and arc length is proportional to the number of sentences
containing the word. (d) Questioner’s emotion distribution pie chart before
observing the hidden image, along with the most affective words for each
specific emotion from the dialogue.

Analyzing Questions.  Fig. 3(a) shows the distribution of the lengths of
questions for AffectVisDial and VisDial (see the solid red lines). The plot
indicates that questions and answers of AffectVisDial are significantly
longer. Fig. 4 (a) shows Sunbursts visualization of the distribution of
questions based on the first four words in the dataset. _Binary Questions vs
Binary Answers:_ It is interesting to see that the top 10 answers are binary
answers with details, e.g., “No there are no trees or plants in the image” or
“Yes there is a man in the image.” This is a consequence of the questioner not
being able to see the image. Thus, they are asking contextually relevant
questions like “Do you see any trees or plants in the image?” in an attempt to
explore the scene. Das et al. [15] observed that the VQA dataset contains
binary questions that can be answered by simply “yes”/“no”/“maybe”. Regarding
answers which contain “yes”/“no” as binary answers, they reported that only
46.96% of all “yes”/“no” responses were binary answers containing “yes” in
VisDial. However, in VQA there was a bias towards “yes”, with 61.40% of binary
responses being “yes”. In our dataset, there is a balance between binary
questions containing “yes” and “no”, 50.32% and 49.68%, respectively. The
percentage of “yes”/“no” questions with respect to the total number of answers
from the training set is 52.7% for VisDial and for 25.5% for AffectVisDial
($>$ 50% reduction).

Figure 5: Top: percentage of change of emotion before/after the Questioner
observes the image in the dataset. Bottom: emotion distribution across
continents.

Analyzing Answers. Answers in AffectVisDial are much longer and more
informative - average length 14.85 words; see Tab. 1 and the solid green and
dashed green plots in Fig. 3(a). The top-1000 frequent answers in VisDial [15]
cover 63% of all answers, while for AffectVisDial they cover only 38% . Since
the number of dialogs in VisDial is higher than AffectVisDial , we sampled a
subset of dialogs of the same size as our whole dataset size and computed the
percentage coverage of answers on this subset from VisDial. Fig. 3(b) shows
the percent coverage of unique answers overall answers in the training set.
Fig. 4 (b) visualize answer distribution based on the starting few words. A
large portion of answers are descriptive to convey the existence of objects in
the image (e.g., “there are/is” or “there seems like”), and many answers start
with“there are” or “is” since questioners are querying whether things are in
the image.

Analyzing emotion explanations. We report three types of analysis for
explanations: explanations of the Questioner (1) before and (2) after the
image observation, and Answerer explanations. Tab. 1 shows that on average
explanations have lengths 27.29 and 28.95 for Questioner and Answerer,
respectively. Tab. 2 shows that the average number of PoS from explanations
before observing images is larger than those after observing the image.
Answerer explanation contains slightly more pronouns, so we speculate that
they can easily refer to visual cues since they can see the hidden image.

Emotion Distribution. We report emotion distribution before observing the
image in Fig. 4 (d) with the top words associated with each emotion. It is
obvious that the most dominant emotion is “contentment”. Fig. 5 (top) shows
the percentage of Questioner, who changed their emotion choice after observing
the hidden image. On average, 23.22% of the time, annotators changed their
minds after looking at the hidden image. In general, dialog information is
informative enough for the Questioner to conclude their final emotional
responses. The emotion category that was influenced the most by the emotion
change is “something else”, where the Questioner may not have to find enough
linguistic details associated with images. Fig. 5 (bottom) shows the
distribution of emotions before observing the image across continents.
“contentment” is the most dominant emotion across annotators from all
continents. Interestingly, we can observe that there are some similarities in
the emotion distribution between North America and Europe: the emotion labels
like “contentment”, “excitement”, and“fear” are almost identical between these
groups.

##  4 Task and Neural Baselines

The Affective Visual Dialog task involves two subtasks, dialog-based question
answering 4.1 and dialog-based emotion classification and affective
explanation generation 4.2.

Notations: To begin with, we first elaborate on several notations.
Specifically, $I$ represents an image, and two emotion labels (in the form of
positive-negative pairs), denoted as $E$, and associated opinions as $C$.
Moreover, a dialog $D$ containing 10 pairs of questions and answers about the
image can be represented as $D=\\{(Q_{1},A_{1}),...,(Q_{10},A_{10})\\}$. At
any given round $t$, a context $H_{t}$ is defined as the combination of the
image, emotion labels, captions, and the dialog history up to that point,
which can be represented as
$H_{t}=\\{I,C,(Q_{1},A_{1}),...,(Q_{t-1},A_{t-1})\\}$.

###  4.1 Dialog-based Question Answering Subtask

In this subtask, neural systems’ aim is to generate an answer $A_{t}$ for a
given question $Q_{t}$ and context $H_{t}$ at round $t$, similar to [15]. Two
approaches can be used: generative [41, 11, 40] and discriminative [11, 39,
41]. Generative models generate an answer, while discriminative models rank
pre-defined candidate answers.

Neural Baselines: We explore several neural models for this task. Firstly, we
utilize the simple Nearest Neighbors approach, as introduced in Das et al.
[15]. We experiment with two variants: NN-Q, which finds $k$ similar questions
based on GloVE [43] embeddings and takes the average similarity scores between
candidates and $k$ answers for ranking options; and NN-QI, which extracts $K$
similar questions and then selects a subset of $k$ questions based on image
similarity features from VGG-16 [52]. Additionally, we evaluate two state-of-
the-art models: VisDial-BERT [39], which adopts VilBERT [34] pretrained on
Conceptual Captions [50] and VQA [7] and is fine-tuned on Visual Dialog [15]
tasks; and LTMI [41], a SOTA model that can be used as both a generative and
discriminative model.

###  4.2 Affective Explanation Generation Subtask

In this subtask, we investigate the ability of neural models to generate
emotion explanations given $D$ and, optionally $I$. Additionally, we
incorporate $E$ and $C$ to provide a comprehensive context for the generation
process. To simultaneously generate the emotion class and its corresponding
explanation for both the questioner and answerer, we experiment with different
generative language models. To make the explanation more faithful to the
identified emotion, we design a prompt “I feel EMOTION because EXPLANATION”
and train the models with this prompt. Generating emotion and explanation
together helps the model understand the reasoning behind the emotion and
produces more accurate predictions.

Neural Baselines: We consider two generative text models (BART [28], T5-large
[44]) and a recently introduced multi-modal model NLX-GPT [48] for emotion and
explanation generation for both Questioner and Answerer. Since the Answerer
has always access to the image $I$, we include $I$ in the form of text from
pretrained BLIP [29] model for text models T-5 and BART. For the Questioner,
we experiment with two variants: the model (1) w/o and (2) w/ access to the
image $I$. For (2), we also include $I$ in the input similar to the Answerer
case and predict the emotion and explanation of the Questioner after observing
the image. We fine-tune the NLX-GPT model [48] for generating emotional
explanations, which can accept both visual and language inputs and produce
answers and explanations. We train the model from the perspectives of both the
Questioner and the Answerer using the same input as the previously mentioned
language models. The image $I$ is passed through the model’s image encoder
before being fed to the decoder for final emotion and explanation generation
(see the supplementary for implementation details).

###  4.3 Dialog-based Emotion Prediction Task

We propose a dialog-based classification task that aims to predict the emotion
label based on a visually grounded conversation, which can be formulated as a
standard document classification problem with 9 emotion categories. To address
this task, we utilize a pre-trained RoBERTa [33] model and fine-tune it on our
dataset.

Model R@1($\uparrow$) R@5($\uparrow$) R@10($\uparrow$) MRR($\downarrow$)
MR($\downarrow$) NN-Q 0.11 0.26 0.38 0.65 32.01 NN-QI 0.07 0.16 0.31 0.71
36.32 LTMI-G [41] 0.18 0.25 0.55 0.36 24.47 LTMI-D [41] 0.25 0.39 0.84 0.58
5.62 VisDial-BERT [39] 0.23 0.35 0.79 0.47 6.10

Table 3: Results on dialog-based question answering task.

##  5 Experimental Results

Evaluation Metrics. For the Dialog-based Question Answering task, we use
standard retrieval-based metrics introduced in [15]: mean rank (MR), mean
reciprocal rank (MRR), and recall R@$k$ ($k=1,5,10$) based on 100 candidate
answers. Candidates include a ground truth answer, 50 possible answers, 30
popular answers, and 19 random answers. To evaluate explanation generation
quality and linguistic similarity to human explanations, we use popular
machine-based captioning metrics: BLEU [42], BERT-score [60], and BART-score
[58]. We evaluate predicted emotion categories using a weighted F1-score due
to data imbalance over the 9 emotion classes.

Dialog-based Classification. After fine-tuning our RoBERTa-based classifier,
we observe that it achieves 62.2% accuracy. Note that since emotions are
subjective in nature, their prediction might be a challenging task.

Dialog-based Question Answering. Tab. 3 shows the results of the dialog-based
question-answering subtask, where LTMI-D achieves the highest scores among all
baselines in terms of retrieval metrics. Naive neighboring approaches perform
poorly, possibly due to the large number of unique answers compared to
questions. Discriminative methods such as VisDial-BERT and LTMI-D outperform
the generative LTMI-G on average by 0.06, 0.12, and 0.27 in R@1, R@5, and R@10
metrics, respectively. The results suggest that the task is challenging in its
open form, and generative models may not be as effective as discriminative
models.

Model $I$ $E$ $C$ $D$ BLEU($\uparrow$) BERT($\uparrow$) BART($\downarrow$)
Emo-F1($\uparrow$) NLX-GPT [48] $\times$ ✓ ✓ $\times$ 0.09 0.65 -6.67 29.55
NLX-GPT [48] $\times$ ✓ ✓ ✓ 0.23 0.86 -5.05 46.17 1-9[1.5pt/2pt] NLX-GPT [48]
✓ ✓ ✓ $\times$ 0.11 0.70 -5.72 35.50 NLX-GPT [48] ✓ ✓ ✓ ✓ 0.27 0.88 -4.92
43.61 BART-Large [28] $\times$ ✓ ✓ $\times$ 0.02 0.02 -5.71 2.10 BART-Large
[28] $\times$ $\times$ ✓ ✓ 0.26 0.88 -4.25 47.49 BART-Large [28] $\times$ ✓ ✓
✓ 0.27 0.88 -4.26 51.62 1-9[1.5pt/2pt] BART-Large [28] ✓ ✓ ✓ $\times$ 0.04
0.85 -5.38 38.11 BART-Large [28] ✓ $\times$ ✓ ✓ 0.27 0.88 -4.18 45.52 BART-
Large [28] ✓ ✓ ✓ ✓ 0.28 0.89 -4.18 50.43 T5-Large [44] $\times$ ✓ ✓ $\times$
0.05 0.05 -5.98 15.20 T5-Large [44] $\times$ $\times$ ✓ ✓ 0.25 0.88 -4.54
40.48 T5-Large [44] $\times$ ✓ ✓ ✓ 0.26 0.87 -4.4 46.34 1-9[1.5pt/2pt]
T5-Large [44] ✓ ✓ ✓ $\times$ 0.04 0.85 -5.38 38.11 T5-Large [44] ✓ $\times$ ✓
✓ 0.28 0.88 -4.35 41.91 T5-Large [44] ✓ ✓ ✓ ✓ 0.28 0.89 -4.34 42.95

Table 4: Results on emotion and explanation generation setup for Questioner.
$I,E,C,D$ represents the image, 2 opposed emotion labels, associated opinions,
and the dialog defined in Sec. 4.

Questioner Explanation Generation. Table 4 presents the results for the
explanation generation subtask. It is evident that models relying solely on
opinions ($C$) and emotion representations ($E$) as input exhibit
significantly poorer performance across all metrics compared to other
baselines. For instance, the BART model without dialogue ($D$) underperforms
its counterpart with dialogue by 0.24 BLEU scores, indicating the
insufficiency of relying solely on two input opinions. Incorporating textual
descriptions of images ($I$) for text-based models leads to noteworthy
improvements, as they provide valuable information about the visual input.
Notably, the T5 model demonstrates considerable enhancements in metrics such
as BLEU, BERT, and BART when exposed to image textual descriptions during
training and testing. Intriguingly, when dialogues ($D$) are introduced as an
additional input for these models, their performance significantly improves
across all metrics. For instance, T5-Large and NLX-GPT which incorporate $D$
outperform those that do not take $D$ as input by 0.24 and 0.18 BLEU scores,
respectively, further underscoring the critical role of dialogue in guiding
the model’s emotion-related tasks. The results for Answerer are available in
the supplementary.

Figure 6:  Generated Questioner’s Explanation from our T5-Large baseline.

Human Studies. To validate our model’s effectiveness, we conducted studies on
Amazon Mechanical Turk. We asked participants to evaluate the reasonableness
of our dialog-based QA task and conducted a Turing test for emotion
explanation generation. Results indicate that over 90% of produced answers
were considered reasonable and 55% of explanations were considered human-like,
demonstrating the effectiveness of our approach in capturing emotional
reasoning in visually grounded conversations (Details in supplementary).

Fig. 6 shows the generated emotion and corresponding explanation from the
Questioner. The output closely resembles human explanations (more examples are
in the supplementary).

Emotion Guided Answer Generation. In this study, we investigate the influence
of dialogue answers on resulting emotions. To achieve this, we manipulate the
answers to elicit specific emotional reactions. For each question in a
dialogue, we select the answer candidate from similar questions that maximizes
the probability of the target emotion, as determined by our pretrained
RoBERTa-based emotion classifier [33]. The initial two or three turns of the
dialogue remain unchanged, while subsequent answers are replaced by the
selected candidate answers. Fig. 8 shows how gradually altering the answers
can shift the original emotion towards an opposing direction, illustrating the
role of linguistic details in driving emotions (e.g., the absence of trees
increases the probability score of ”fear”).

Figure 7: Example of altering answers to evoke opposite emotions, and make
”edit” in the original image.

Towards Emotionally Reasoned Image Editing. While the above study has provided
valuable insights, it is essential to acknowledge certain limitations in the
methodology. Firstly, emotions are complex and subjective, and the
classifier’s output may not always align perfectly with true human emotional
experiences. Furthermore, the chosen answers may not necessarily be grounded
in the visual input. An interesting avenue for exploration is editing initial
visual input such that it is aligned with the candidate’s answers. This
approach could be valuable, for example, for emotion-driven content creation.
To illustrate this, we utilized our roBERTa-based classifier to select an
answer for a given question to increase the probability score of the target
emotion. Subsequently, we used an image editing model [9] to modify the
original image accordingly (see Fig. 7). In this example, the suggested
practice reasoned that the contentment emotion of the visual stimuli can be
increased by suggesting to include a mountain in the image. This approach
paves the way for the creation of collaborative image editing systems capable
of offering emotionally informed suggestions for edits.

Figure 8: Examples of altering answers to evoke opposite emotions. The left
side of each example shows the original dialog and the right side shows the
replaced answer with the corresponding probability of the target emotion

Emotion Awareness of ChatGPT and GPT-4. We explored the potential of ChatGPT
[1], a foundation model known for its impressive zero-shot question-answering
performance, for predicting emotions and generating corresponding explanations
on our newly proposed dataset. To this end, we employed the latest API [2],
GPT-4, with either $\\{E,C\\}$ or $\\{E,C,D\\}$, along with our designed
instructions, as inputs to enable ChatGPT to generate emotions and their
explanations. The results showed that the Emotion F1 scores were $19.84\%$ and
$26.64\%$, respectively for two input combinations. While additional dialog
input improved the emotion prediction, ChatGPT still struggled to capture
emotional information through conversation and understand artwork descriptions
at a human level. In contrast, our baselines trained on our dataset showed
superior results, yielding more than $40\%$ in F1 score. These limitations
highlight the significance of our dataset in advancing emotion-aware AI
systems for future applications. Further details regarding the instructions
and generated examples are provided in the supplementary material.

##  6 Conclusion

Emotions are an integral part of human nature. A deeper comprehension of how
emotions arise from natural conversation can aid in the development of more
human-compatible machines. In pursuit of this goal, we introduce a novel
dataset, called AffectVisDial , which records the dialogue between two agents
discussing an image and the resultant emotion they construct from the visually
grounded conversation. By adapting and training the state-of-the-art models
using our AffectVisDial dataset, we show the potential of AI systems for
generating affective explanations grounded in dialogues that are similar to
human emotions. We also demonstrated that the models trained on AffectVisDial
can facilitate both emotion-guided answer generation and pave the way toward
building and advancing emotionally reasoned collaborative image editing
systems. We hope that our dataset and models will help ease future research in
affective vision and language.

Acknowledgement. This project is funded by KAUST BAS/1/1685-01-01, SDAIA-KAUST
Center of Excellence in Data Science and Artificial Intelligence. The authors
express their appreciation to Jack Urbanek, Sirojiddin Karimov, and Umid
Nejmatullayev for their valuable assistance in data collection setup. Lastly,
the authors extend their gratitude to the diligent efforts of the Amazon
Mechanical Turkers, DeepenAI, and SmartOne teams, as their contributions were
indispensable for the successful completion of this work.

## References

  * [1] Introducing chatgpt.  https://openai.com/blog/chatgpt, 2023.  Accessed: March 7, 2023.
  * [2] Openai api: Gpt-4.  https://platform.openai.com/docs/models/gpt-4, 2023.  Accessed: July 25, 2023.
  * [3] Adnen Abdessaied, Mihai Bâce, and Andreas Bulling.  Neuro-symbolic visual dialog.  arXiv preprint arXiv:2208.10353, 2022.
  * [4] Panos Achlioptas, Maks Ovsjanikov, Leonidas Guibas, and Sergey Tulyakov.  Affection: Learning affective explanations for real-world visual data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6641–6651, June 2023.
  * [5] Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas.  Artemis: Affective language for visual art.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11569–11579, 2021.
  * [6] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971–4980, 2018.
  * [7] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [8] Dmitry Bogdanov, Alastair Porter, Philip Tovstogan, and Minz Won.  Mediaeval 2019: Emotion and theme recognition in music using jamendo.  In Larson M, Hicks S, Constantin MG, Bischke B, Porter A, Zhao P, Lux M, Cabrera Quiros L, Calandre J, Jones G, editors. MediaEval’19, Multimedia Benchmark Workshop; 2019 Oct 27-30, Sophia Antipolis, France. Aachen: CEUR; 2019. CEUR Workshop Proceedings, 2019.
  * [9] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18392–18402, June 2023.
  * [10] Sven Buechel and Udo Hahn.  Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis.  arXiv preprint arXiv:2205.01996, 2022.
  * [11] Cheng Chen, Zhenshan Tan, Qingrong Cheng, Xin Jiang, Qun Liu, Yudong Zhu, and Xiaodong Gu.  Utc: A unified transformer with inter-task contrastive learning for visual dialog.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18103–18112, 2022.
  * [12] Sheng-Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Lun-Wei Ku, et al.  Emotionlines: An emotion corpus of multi-party conversations.  arXiv preprint arXiv:1802.08379, 2018.
  * [13] Community.  Wiki art.  https://www.wikiart.org/, 2020.  Accessed: 2020-11-06.
  * [14] Alan S Cowen, Xia Fang, Disa Sauter, and Dacher Keltner.  What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures.  Proceedings of the National Academy of Sciences, 117(4):1924–1934, 2020.
  * [15] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra.  Visual dialog.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326–335, 2017.
  * [16] Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville.  Guesswhat?! visual object discovery through multi-modal dialogue.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [17] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi.  Goemotions: A dataset of fine-grained emotions.  arXiv preprint arXiv:2005.00547, 2020.
  * [18] Ed Diener, Christie Napa Scollon, and Richard E Lucas.  The evolving concept of subjective well-being: the multifaceted nature of happiness.  2009\.
  * [19] Paul Ekman.  An argument for basic emotions.  Cognition & emotion, 6(3-4):169–200, 1992.
  * [20] Jianyu Fan, Miles Thorogood, and Philippe Pasquier.  Emo-soundscapes: A dataset for soundscape emotion recognition.  In 2017 Seventh international conference on affective computing and intelligent interaction (ACII), pages 196–201. IEEE, 2017.
  * [21] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The” something something” video database for learning and evaluating visual common sense.  In Proceedings of the IEEE international conference on computer vision, pages 5842–5850, 2017.
  * [22] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [23] Hatice Gunes and Massimo Piccardi.  A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior.  In 18th International conference on pattern recognition (ICPR’06), volume 1, pages 1148–1153. IEEE, 2006.
  * [24] Hsiao-Tzu Hung, Joann Ching, Seungheon Doh, Nabin Kim, Juhan Nam, and Yi-Hsuan Yang.  Emopia: a multi-modal pop piano dataset for emotion recognition and emotion-based music generation.  arXiv preprint arXiv:2108.01374, 2021.
  * [25] Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler.  What are you talking about? text-to-image coreference.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3558–3565, 2014.
  * [26] Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach.  Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog.  arXiv preprint arXiv:1903.03166, 2019.
  * [27] Vicki R LeBlanc, Meghan M McConnell, and Sandra D Monteiro.  Predictable chaos: a review of the effects of emotions on attention, memory and decision making.  Advances in Health Sciences Education, 20(1):265–282, 2015.
  * [28] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, 2020.
  * [29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [30] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu.  Dailydialog: A manually labelled multi-turn dialogue dataset.  arXiv preprint arXiv:1710.03957, 2017.
  * [31] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [32] Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, and Guoying Zhao.  imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10631–10642, 2021.
  * [33] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [34] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [35] Jana Machajdik and Allan Hanbury.  Affective image classification using features inspired by psychology and art theory.  In Proceedings of the 18th ACM international conference on Multimedia, pages 83–92, 2010.
  * [36] Youssef Mohamed, Mohamed Abdelfattah, Shyma Alhuwaider, Feifan Li, Xiangliang Zhang, Kenneth Church, and Mohamed Elhoseiny.  Artelingo: A million emotion annotations of wikiart with emphasis on diversity over language and culture.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8770–8785, 2022.
  * [37] Youssef Mohamed, Faizan Farooq Khan, Kilichbek Haydarov, and Mohamed Elhoseiny.  It is okay to not be okay: Overcoming emotional bias in affective image captioning by contrastive data collection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21263–21272, 2022.
  * [38] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 10(1):18–31, 2017.
  * [39] Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das.  Large-scale pretraining for visual dialog: A simple state-of-the-art baseline.  In European Conference on Computer Vision, pages 336–352. Springer, 2020.
  * [40] Vishvak Murahari, Prithvijit Chattopadhyay, Dhruv Batra, Devi Parikh, and Abhishek Das.  Improving generative visual dialog by answering diverse questions.  arXiv preprint arXiv:1909.10470, 2019.
  * [41] Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani.  Efficient attention mechanism for visual dialog that can handle all the interactions between multiple inputs.  In European Conference on Computer Vision, pages 223–240. Springer, 2020.
  * [42] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [43] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21(140):1–67, 2020.
  * [45] Hiranmayi Ranganathan, Shayok Chakraborty, and Sethuraman Panchanathan.  Multimodal emotion recognition using deep learning architectures.  In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–9. IEEE, 2016.
  * [46] Anna Rohrbach, Marcus Rohrbach, Siyu Tang, Seong Joon Oh, and Bernt Schiele.  Generating descriptions with grounded and co-referenced people.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4979–4989, 2017.
  * [47] Stuart Russell.  Human compatible: Artificial intelligence and the problem of control.  Penguin, 2019.
  * [48] Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis.  Nlx-gpt: A model for natural language explanations in vision and vision-language tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8322–8332, 2022.
  * [49] Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal.  Visual reference resolution using attention memory for visual dialog.  Advances in neural information processing systems, 30, 2017.
  * [50] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of ACL, 2018.
  * [51] Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston.  Image chat: Engaging grounded conversations.  arXiv preprint arXiv:1811.00945, 2018.
  * [52] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [53] Carlo Strapparava and Rada Mihalcea.  Semeval-2007 task 14: Affective text.  In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70–74, 2007.
  * [54] Jack Urbanek and Pratik Ringshia.  Mephisto: A framework for portable, reproducible, and iterative crowdsourcing, 2023.
  * [55] Victoria Yanulevskaya, Jan C van Gemert, Katharina Roth, Ann-Katrin Herbold, Nicu Sebe, and Jan-Mark Geusebroek.  Emotional valence categorization using holistic image features.  In 2008 15th IEEE international conference on Image Processing, pages 101–104. IEEE, 2008.
  * [56] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang.  Building a large scale dataset for image emotion recognition: The fine print and the benchmark.  In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  * [57] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  * [58] Weizhe Yuan, Graham Neubig, and Pengfei Liu.  Bartscore: Evaluating generated text as text generation.  Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
  * [59] Jonathan R Zadra and Gerald L Clore.  Emotion and perception: The role of affective information.  Wiley interdisciplinary reviews: cognitive science, 2(6):676–685, 2011.
  * [60] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  In International Conference on Learning Representations, 2019.


#  Ultrathin carpet cloak enabled by infinitely anisotropic medium

Mohammad Hosein Fakheri  Ali Abdolali<EMAIL_ADDRESS>Applied
Electromagnetic Laboratory, School of Electrical Engineering, Iran University
of Science and Technology, Tehran, 1684613114, Iran.

###### Abstract

Thanks to the pioneering studies conducted on the fields of transformation
optics (TO) and metasurfaces, many unprecedented devices such as invisibility
cloaks have been recently realized. However, each of these methods has some
drawbacks limiting the applicability of the designed devices for real-life
scenarios. For instance, TO studies lead to bulky coating layer with the
thickness that is comparable to, or even larger than the dimension of the
concealed object. In this paper, based on the coordinate transformation, an
ultrathin carpet cloak is proposed to hide objects with arbitrary shape and
size using a thin anisotropic material, called as infinitely anisotropic
medium (IAM). It is shown that unlike the previous metasurface-based carpet
cloaks, the proposed IAM hides objects from all viewing incident angles while
it is extremely thin compared with the object dimensions. This material also
circumvents the conventional transformation optics’ complexities and could be
easily implemented in practical scenarios. To demonstrate the capability of
the proposed carpet cloak, several full-wave simulations are carried out.
Finally, as a proof of concept, the IAM is implemented based on the effective
medium theory which exhibits good agreement with the results obtained from the
theoretical investigations. The introduced material not only constitutes a
significant step towards the invisibility cloak but also can greatly promote
the practical application of the other TO-based devices.

An invisibility cloak that can conceal objects from the electromagnetic (EM)
wave has attracted considerable attentions for centuries, but its realization
remained unattainable until the advent of metamaterials. To design such an
unprecedented device, various techniques such as, scattering cancelation [1]
and transformation optics [2], have been proposed. However, the method of
scattering cancelation is limited to small hidden objects in comparison to the
operative wavelength [1]. From the viewpoint of TO and based on the form
invariance of Maxwell’s equations, an initial coordinate system (_i.e.,_
virtual domain) is transformed to its counterpart in another arbitrary
coordinate system (_i.e.,_ physical domain), which results in a direct link
between the material properties and metric tensor of the transformed space
having the desired EM functionalities [3, 4, 5, 6]. However, the materials
obtained through this approach are inhomogeneous and anisotropic, which are
difficult to be implemented. To obviate the anisotropy issue, the method of
quasi-conformal mapping is utilized [7]. Nevertheless, even by performing such
a simplifying assumption, this cloak causes a lateral shift of the scattered
wave, whose value is comparable to the height of the cloaked object, making
the object detectable [8]. Moreover, the mentioned techniques generally yield
a bulky coating later with a thickness comparable to, or even larger than the
dimension of the concealed object. As the 2D version of metamaterials,
metasurfaces can control the EM waves via imparting abrupt phase changes
determined by subwavelength elements at the interface and have overcame the
certain drawbacks of bulky metamaterials [9, 10, 11]. These ultrathin
structures have paved the way towards achieving carpet cloaks hiding objects
on the ground plane[12, 13]. Despite claims about wide-angle metasurface
carpet cloak [14], their functionality is restricted to a specific angular
range. Indeed, the cloaking performance is drastically deteriorated out of
these incident wave angles. Consequently, for the above-mentioned reasons, one
must ask the question of whether there is any alternative way to conceal an
object with an ultrathin carpet cloak, regardless of the incident wave angle.

Here, we successfully design an ultrathin all-angle coating layer to conceal
objects with arbitrary size and shape located on the ground, based on a
special type of material named infinitely anisotropic medium (IAM). It will be
demonstrated that the introduced IAM is independent of the concealed object
geometry in such a way that if the object shape is changed, one can still use
the same IAM. Several numerical simulations are carried out to verify the
validity of the propounded invisibility cloak. It is observed that numerical
results are in good agreement with the theoretical investigations indicating
the validity of the presented approach. As a proof of concept, an ultrathin
carpet cloak is realized through the effective medium theory (EMT) for which,
the spiral resonator (SR)-meander line metamaterial blocks have been
elaborately engineered to play the role of the required IAM.

Figure 1:  (a) Schematic view of the proposed ultrathin carpet cloak for
arbitrary shape object. (b) Designing a carpet cloak based on coordinate
transformation. a cream region in the virtual space is mapped to a brown
region in the physical space.

Fig. 1(a) shows the schematic view of the proposed carpet cloak that conceals
the arbitrary shape object with an ultrathin coating layer. Let us start with
a straightforward coordinate transformation in the general form. In order to
conceal an arbitrary shape object with the curve of $y_{1}(x)$, the schematic
of Fig. 1(b) is used as the space transformation. As can be seen, a cream
region enclosed between the ground $\left(y=0\right)$ and curve $y_{2}(x)$ in
the virtual space is compressed into the brown region encircled by $y_{1}(x)$
and $y_{2}(x)$ in the physical space. Indeed, the ground is transformed to the
concealed object boundary with the curve $y_{1}=f(x)$ while the boundary
$y_{2}=\Gamma f(x)$ is mapped to itself keeping the exterior EM fields
unchanged.

The transformation function of such a mapping could be written as
$x^{\prime}=x$, $y^{\prime}=\alpha y+f(x)$ and $z^{\prime}=z$ where
$\alpha=\left(\Gamma-1\right)/\Gamma$. According to the coordinate
transformation theory, when a space $(x,y,z)$ is transformed into another
space $(x^{\prime},y^{\prime},z^{\prime})$ of different shape and size, the
permittivity $\overline{\overline{{{\varepsilon}^{\prime}}}}$ and permeability
$\overline{\overline{{{\mu}^{\prime}}}}$ values in the transformed space are
given by
$\overline{\overline{{{\varepsilon}^{\prime}}}}={\Lambda\varepsilon{{\Lambda}^{T}}}/{\det\Lambda}\;$
and
$\overline{\overline{{{\mu}^{\prime}}}}={\Lambda\mu{{\Lambda}^{T}}}/{\det\Lambda}\;$,
where $\varepsilon$ and $\mu$ refer to the permittivity and permeability in
the original space and
$\Lambda={\partial\left({x}^{\prime},{y}^{\prime},{z}^{\prime}\right)}/{\partial\left(x,y,z\right)}\;$
indicates the Jacobian transformation tensor [2]. Consequently, the
constitutive parameters of the proposed carpet cloak could be expressed as

$\frac{\overline{\overline{\varepsilon^{\prime}}}}{\varepsilon}=\frac{\overline{\overline{\mu^{\prime}}}}{\mu}=\frac{1}{\alpha}\begin{bmatrix}\
1&f^{\prime}(x)&0\\\ f^{\prime}(x)&\alpha^{2}+(f^{\prime}(x))^{2}&0\\\
0&0&1\end{bmatrix}$ (1)

The obtained material of Eq.(1) is inhomogeneous and anisotropic with off-
diagonal components of $f^{\prime}(x)$, which may cause serious difficulties
in its realization process. However, as we aim to design an ultrathin carpet
cloak, we set $\Gamma\rightarrow 1$ which leads to $\alpha\rightarrow 0$. The
diagonal form of Eq.(1) is obtained by rotating the matrix around its
principle axis at the angle of $\varphi=tan^{-1}(f^{\prime}(x))+\pi/2$ [15].
Therefore, the diagonal form of Eq.(1) is expressed as

$\frac{\overline{\overline{\varepsilon^{\prime}_{d}}}}{\varepsilon}=\frac{\overline{\overline{\mu^{\prime}_{d}}}}{\mu}=\begin{bmatrix}\
0&0&0\\\ 0&\frac{1+(f^{\prime}(x))^{2}}{\alpha}&0\\\
0&0&\frac{1}{\alpha}\end{bmatrix}\overset{\alpha\to
0}{\mathop{=}}\,\begin{bmatrix}\ 0&0&0\\\ 0&\infty&0\\\
0&0&\infty\end{bmatrix}$ (2)

The material achieved by Eq. (2) is called an IAM with the main axis along the
$x^{\prime}$ direction. Therefore, by considering Eq.(2) and the rotation
angle $\varphi$, one can conclude that an ideal ultrathin carpet cloak is
composed of the IAM whose main axis is perpendicular to the object’s surface.

Figure 2:  Performance of the invisibility carpet cloak based on IAM for a
triangle shape object with $\alpha=0.01$. (a) Magnetic field distribution at
$\phi_{inc}=135^{\circ}$ . (b) Far-field patterns at $\phi_{inc}=110^{\circ}$,
$\phi_{inc}=135^{\circ}$ and $\phi_{inc}=160^{\circ}$.

In the following, we carry out full-wave simulations using COMSOL Multiphysics
finite element solver to demonstrate the functionality of ultrathin carpet
cloaks to conceal arbitrary shape objects from all viewing angles. We assume
the structure is illuminated by a TM-polarized ($H$ along the $z$ direction)
plane wave at $f$=3.5 GHz. The first example is dedicated to a triangle object
with the base and height parameters of $0.2m$ and $0.4m$, respectively. Since,
selecting the exact value of 0 for $\alpha$ causes some errors in the
simulation process, we set $\alpha=0.01$. Fig. 2(a) illustrates the near field
distributions of the magnetic field where the object is covered with the
proposed IAM. As can be seen, under the incident wave angle of
$\phi_{inc}=135^{\circ}$, the object becomes invisible, where the near field
distribution mimics the specular reflection behavior of a flat ground plane.
To further demonstrate the effectiveness of the proposed ultrathin cloak, the
structure is examined under two other incident wave angles
$\phi_{inc}=110^{\circ}$ and $\phi_{inc}=160^{\circ}$. The scattering cross
section of three mentioned cases are plotted in Fig. 2(b), where a single beam
is noticed at the desired angle for each case. The difference of their beam
width comes from the fact that the more oblique the incident wave, the smaller
the equivalent aperture size [16]. Consequently,the presented IAM can hide
objects from all range of incident angles while having an extremely thin
profile in comparison to the object size.

Moreover, for the objects with arbitrary shape that are important in the
practical scenarios, the previously reported approaches are not applicable due
to the necessity to inhomogeneous and shape-dependent materials. Nevertheless,
the material introduced in Eq.(2), can mitigate most of these drawbacks. To
this aim, an arbitrary shape object is coated by a thin IAM layer with
$\alpha=0.03$. As illustrated in Fig. 3(a), the total magnetic field
distribution of the concealed object is identical to that of the ground plane
as if the object does not exist. Moreover, in contrast to the previous
metasurface cloaks, when the angle of incidence varies, the functionality of
our designed cloak remains unchanged, as validated with the scattering cross
section in Fig. 3(b). In other words, the functionality of the cloak presented
here is not restricted to a specific range of incident wave angles.

Figure 3:  Performance of the invisibility carpet cloak based on IAM for an
arbitrary shape object with $\alpha=0.03$. (a) Magnetic field distribution at
$\phi_{inc}=135^{\circ}$ . (b) Far-field patterns at $\phi_{inc}=110^{\circ}$,
$\phi_{inc}=135^{\circ}$ and $\phi_{inc}=160^{\circ}$.

Although the propounded cloak can conceal the objects with ultrathin coating
layer from all viewing angles, this challenging question is raised that how an
IAM could be realized in practice. To answer this question, we will take the
advantage of EMT so as to validate the performance of an implemented triangle
shape carpet cloak. With the assumption of TM polarization, the only important
parameters are $\varepsilon_{xx}=0$,
$\varepsilon_{yy}={1+(f^{\prime}(x))^{2}}/\alpha$ and $\mu_{zz}=1/\alpha$. As
discussed above, when $\alpha\rightarrow 0$, the IAM with
$\varepsilon=diag[0,\infty]$ and $\mu=\infty$ is acquired where $diag$[·]
represents a diagonal matrix.

Figure 4:  A multi-layered structured that mimics an IAM. The electric
permittivity and magnetic permeability are changed periodically in $y$ and $z$
directions, respectively.  Figure 5:  (a) Composite of SR-meander line. (b)
Side view of the proposed metamaterial. (c) The geometry of SR which controls
the $\mu$ value. (d) The geometry of meander line which controls the
$\varepsilon$ value. (e) The retrieved materials for $\varepsilon=-1$,
$\mu=-1$ with parameters $t_{1}=0.2mm$, $t_{2}=0.1mm$, $d=0.58mm$, $a=1.7mm$,
$l=1.5mm$, $w=0.045mm$ and the SR has four turns, the inner radius of
$0.226mm$ and the gap value is $0.033mm$. (f) The retrieved materials for
$\varepsilon=1$, $\mu=-1$ with parameters $t_{1}=0.2mm$, $t_{2}=0.1mm$,
$d=0.58mm$, $a=2mm$, $l=1.49mm$, $w=0.06mm$ and the SR has four turns, the
inner radius of $0.216mm$ and the gap value is $0.053mm$. (g) The retrieved
materials for $\varepsilon=-1$, $\mu=1$ with parameters $t_{1}=0.2mm$,
$a=1.7mm$, $l=1.54mm$, $w=0.045mm$. Figure 6:  Performance of the realized
carpet cloak for a triangle shape object with $\alpha=0.2$. (a) Magnetic field
distribution at $\phi_{inc}=135^{\circ}$ . (b) Far-field patterns at
$\phi_{inc}=110^{\circ}$, $\phi_{inc}=135^{\circ}$ and
$\phi_{inc}=160^{\circ}$.

According to EMT, the multi-layered structure displayed in Fig. 4 that
consists of four isotropic blocks with constitutive parameters
$\varepsilon_{1,2}$ and $\mu_{1,2}$, can behave as an anisotropic medium with
the parameters of IAM, if the thickness of each layer is much smaller than the
operative wavelength. In general, we have the following effective parameters:

$\displaystyle\varepsilon_{xx}=f\varepsilon_{1}+(1-f)\varepsilon_{2}$ (3)
$\displaystyle\frac{1}{\varepsilon_{yy}}=\frac{f}{\varepsilon_{1}}+\frac{1-f}{\varepsilon_{2}}$
$\displaystyle\frac{1}{\mu_{zz}}=\frac{p}{\mu_{1}}+\frac{1-p}{\mu_{2}}$

in which $f=d_{1}/(d_{1}+d_{2})$ and $p=t_{1}/(t_{1}+t_{2})$. Based on Eq.(3),
if we set $\varepsilon_{1,2}=\pm 1$, $\mu_{1,2}=\pm 1$, $f=0.5$ and $p=0.5$,
then the desired IAM parameters can be attained, where its main axis is
aligned toward $x$-direction. To construct each of these layers, a composite
of SR-meander line array is utilized as depicted in Fig. 5(a). As proof of
principle, a triangle shape carpet cloak with $\alpha=0.2$ is implemented
through the EMT. Since the $\alpha$ value is not too small, the required
constitutive materials are obtained as $\varepsilon_{xx}=0.082$,
$\varepsilon_{yy}=12.08$ and $\mu_{zz}=6$ after the diagonalization. Based on
Eq.(3), one can simply implement this coating layer by using the introduced
structure in Fig. 4 with the parameters $\varepsilon_{1}=1$,
$\varepsilon_{2}=-1$, $\mu_{1}=1$, $\mu_{2}=-1$, $f=0.541$ and $p=0.583$. For
any other shape of concealed objects and $\alpha$ values, the same materials
$(\varepsilon_{1,2}=\pm 1,\mu_{1,2}=\pm 1)$ could be used but with a different
set of the $f$ and $p$ parameters. To achieve the required materials in each
block in Fig. 4, the geometry of SR-meander line unit cells are optimized,
except the brown block which consists of air material or which is filled by
$\varepsilon_{1}=1$ and $\mu_{1}=1$ material. The SRs are designed to have an
adjustable control on $\mu_{z}$ in the cream and orange blocks, which are
sandwiched between two RO3010 substrate
($\varepsilon_{r}=10.2,tg\delta=0.0022$) to reduce the coupling effect between
the adjacent layers (see Fig. 5(b)). In addition, to obtain the same values of
$\varepsilon_{xx}$ and $\varepsilon_{yy}$, two similar meander lines are
printed in both top and bottom sides of the RO3010 substrate, with a
$90^{\circ}$ rotation with respect to each other. The metamaterial building
blocks are designed and simulated with CST Microwave Studio commercial
software, where their retrieved material parameters are plotted in Figs. 5(e),
5(f) and 5(g) for the orange, cream and gray blocks, respectively. As can be
seen, from the retrieved parameters, the designed metamaterials are capable of
possessing the appropriate values at the desired frequency 3.5 GHz.

The triangle shape carpet cloak with $\alpha=0.2$ is realized with the
retrieved parameters of the propounded unit cell (see Fig. 6(a)). The
metamaterial blocks are perpendicular to the object surface which is shown in
the figure inset. As illustrated in Fig. 6(b), the designed metamaterials are
capable of mimicking the IAM behavior and concealing the object from all
viewing angles. It is worth mentioning that regardless of the object geometry,
one can exploit the introduced IAM and achieve the cloaking functionality only
with the four propounded building blocks $(\varepsilon_{1,2}=\pm
1,\mu_{1,2}=\pm 1)$ perpendicular to the object surface and setting the
$f(=d_{1}/(d_{1}+d_{2}))$ values.

In conclusion, in this paper, we have presented a new material based on
coordinate transformation capable of obviating the conventional challenges of
carpet cloaks. The attained material which is called IAM, makes the carpet
cloak extremely thin while the cloaking functionality remain unchanged for all
viewing angles. Numerical simulations are proposed corroborating well the
validity and effectiveness of the propounded approach. In addition, the
metamaterial building blocks that are competent to mimic the behavior of an
IAM are proposed and designed. Therefore, when the geometry of the concealed
object is changed, the same blocks could be used again with different
proportions of the blocks thickness. This fact will, in turn, makes the
proposed material as a good candidate to be used in real-life scenarios. Then,
as a proof of concept, a triangle shape carpet cloak was realized with the aid
of EMT and the designed metamaterials. It was observed that the simulation
results exhibit good agreement with the theoretical predictions, which
corroborates the correctness of the designed procedure. We believe that the
newly proposed material in this paper constitutes a significant step towards
the invisibility cloaks.

## References

  * [1] Alù, Andrea, and Nader Engheta. ”Achieving transparency with plasmonic and metamaterial coatings.” Physical Review E 72, no. 1 (2005): 016623.
  * [2] Pendry, John B., David Schurig, and David R. Smith. ”Controlling electromagnetic fields.” science 312, no. 5781 (2006): 1780-1782.
  * [3] Kwon, Do-Hoon, and Douglas H. Werner. ”Polarization splitter and polarization rotator designs based on transformation optics.” Optics Express 16, no. 23 (2008): 18731-18738.
  * [4] Abdolali, Ali, Hooman Barati Sedeh, and Mohammad Hosein Fakheri. ”Geometry free materials enabled by transformation optics for enhancing the intensity of electromagnetic waves in an arbitrary domain.” Journal of Applied Physics 127, no. 5 (2020): 054902.
  * [5] Fakheri, Mohammad Hosein, and Ali Abdolali. ”Angularly Dispersionless Scattering Patterns for Impenetrable Surfaces: A Straightforward Design Based on Transformation Optics.” IEEE Antennas and Propagation Magazine (2020).
  * [6] Lai, Yun, Jack Ng, HuanYang Chen, DeZhuan Han, JunJun Xiao, Zhao-Qing Zhang, and Che Ting Chan. ”Illusion optics: the optical transformation of an object into another object.” Physical review letters 102, no. 25 (2009): 253902.
  * [7] Li, Jensen, and John B. Pendry. ”Hiding under the carpet: a new strategy for cloaking.” Physical review letters 101, no. 20 (2008): 203901.
  * [8] Zhang, Baile, Tucker Chan, and Bae-Ian Wu. ”Lateral shift makes a ground-plane cloak detectable.” Physical review letters 104, no. 23 (2010): 233903.
  * [9] Yu, Nanfang, Patrice Genevet, Mikhail A. Kats, Francesco Aieta, Jean-Philippe Tetienne, Federico Capasso, and Zeno Gaburro. ”Light propagation with phase discontinuities: generalized laws of reflection and refraction.” science 334, no. 6054 (2011): 333-337.
  * [10] Behroozinia, Sahar, Hamid Rajabalipanah, and Ali Abdolali. ”Real-time terahertz wave channeling via multifunctional metagratings: a sparse array of all-graphene scatterers.” Optics Letters 45, no. 4 (2020): 795-798.
  * [11] Rajabalipanah, Hamid, Kasra Rouhi, Ali Abdolali, Shahid Iqbal, Lei Zhang, and Shuo Liu. ”Real-time terahertz meta-cryptography using polarization-multiplexed graphene-based computer-generated holograms.” Nanophotonics 9, no. 9 (2020): 2861-2877.
  * [12] Zhang, Jing, Zhong Lei Mei, Wan Ru Zhang, Fan Yang, and Tie Jun Cui. ”An ultrathin directional carpet cloak based on generalized Snell’s law.” Applied Physics Letters 103, no. 15 (2013): 151115.
  * [13] Huang, Yijia, Mingbo Pu, Fei Zhang, Jun Luo, Xiong Li, Xiaoliang Ma, and Xiangang Luo. ”Broadband functional metasurfaces: achieving nonlinear phase generation toward achromatic surface cloaking and lensing.” Advanced Optical Materials 7, no. 7 (2019): 1801480.
  * [14] Jiang, Zijie, Qingxuan Liang, Zhaohui Li, Tianning Chen, Dichen Li, and Yang Hao. ”A 3D Carpet Cloak with Non‐Euclidean Metasurfaces.” Advanced Optical Materials 8, no. 19 (2020): 2000827.
  * [15] Barati, H., M. H. Fakheri, and A. Abdolali. ”Experimental demonstration of metamaterial-assisted antenna beam deflection through folded transformation optics.” Journal of Optics 20, no. 8 (2018): 085101.
  * [16] Balanis, Constantine A. Advanced engineering electromagnetics. John Wiley and Sons, 1999.