[1,2]Gulsum Yigit These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Department of Computer Engineering, Kadir Has University, Istanbul, Turkey
2]Department of Computer Engineering, Yildiz Technical University, Istanbul,
Turkey

# From Text to Multimodal: A Comprehensive Survey of Adversarial Example
Generation in Question Answering Systems

<EMAIL_ADDRESS>Mehmet Fatih Amasyali<EMAIL_ADDRESS>*  [

###### Abstract

Integrating adversarial machine learning with Question Answering (QA) systems
has emerged as a critical area for understanding the vulnerabilities and
robustness of these systems. This article aims to comprehensively review
adversarial example-generation techniques in the QA field, including textual
and multimodal contexts. We examine the techniques employed through systematic
categorization, providing a comprehensive, structured review. Beginning with
an overview of traditional QA models, we traverse the adversarial example
generation by exploring rule-based perturbations and advanced generative
models. We then extend our research to include multimodal QA systems, analyze
them across various methods, and examine generative models, seq2seq
architectures, and hybrid methodologies. Our research grows to different
defense strategies, adversarial datasets, and evaluation metrics and
illustrates the comprehensive literature on adversarial QA. Finally, the paper
considers the future landscape of adversarial question generation,
highlighting potential research directions that can advance textual and
multimodal QA systems in the context of adversarial challenges.

###### keywords:

question answering, adversarial question generation, visual question
generation, adversarial datasets, adversarial evaluation metrics

##  1 Introduction

In recent years, natural language processing (NLP) has experienced a
remarkable transformation through revolutions in deep learning architectures
and the availability of vast amounts of textual data. These advancements have
led to the development of sophisticated question-answering (QA) systems that
aim to bridge the gap between human language and machine understanding. State-
of-the-art QA models, such as transformer-based architectures like BERT
(Bidirectional Encoder Representations from Transformers) and its variants,
have demonstrated remarkable performance improvements. These models utilize
pre-trained language models to capture complex linguistic patterns and
contextual dependencies, enabling them to generate accurate and relevant
answers to user questions. Moreover, integrating multimodal data, where
textual, visual, or audio information joins, has also raised the capabilities
of QA systems. Such multimodal QA models, leveraging textual, visual, or audio
signals, have shown promising results in analyzing and generating questions.
However, a vital vulnerability has arisen in the form of adversarial examples
among their outstanding improvements.

Initially presented in computer vision, adversarial examples are meticulously
formulated inputs created to fool machine learning models [1]. These inputs
are modified in malicious forms, almost invisible to humans but are
influential in yielding AI systems to develop inaccurate outcomes. Recent
research has brought to light the utilization of adversarial examples in
various scenarios. To illustrate, adversaries can simulate physical
adversarial examples, thereby confusing autonomous vehicles by manipulating
the appearance of a stop sign within a traffic sign recognition system [2, 3].
Additionally, malicious actors show expertise in generating adversarial
commands aimed at sabotaging automatic speech recognition systems and voice-
controllable systems [4, 5], including crucial platforms such as Apple Siri,
Amazon Alexa, etc.

The influence of adversarial examples quickly attracted the researchers’
attention, defense mechanisms started to develop, and researchers investigated
the underlying causes of the vulnerabilities. With the increased interest in
QA systems, the research has shifted to innovative approaches, such as
adversarial question generation models, to enhance the quality, diversity, and
effectiveness of question generation across textual and multimodal contexts.

Adversarial examples in QA systems are maliciously formulated inputs created
to fool the model and generate inaccurate answers. These adversarially
designed inputs are constructed by applying subtle transformations to the
original question, usually in a form that is invisible to humans but
drastically modifies the model’s generated answer. These adversarial
perturbations could vary from word substitutions to more complicated
linguistic structures aimed at making inaccurate or biased texts, images, or
videos, challenging the robustness of textual and multimodal QA systems.

The impact of adversarial examples on QA systems performance can be critical.
They pose severe challenges to the reliability and security of these systems.
Adversarial examples in QA systems can show several damaging effects:

  * •

Decreased Accuracy: Adversarial examples can reduce the accuracy of QA
systems. By utilizing vulnerabilities in the model’s interpretation of
language and reasoning, these malicious inputs can let the model generate
inaccurate answers, leading to possible misinformation and degraded user
experience.

  * •

Inconsistent Answers: Adversarial examples can also result in inconsistency. A
QA system might provide distinct answers for the same question depending on
malicious perturbations.

Table 1 compares our review and key existing works in question generation in
QA systems. The studies traverse different years and topics, from question
generation in the education domain and multiple-choice question generation to
visual question generation. On the other hand, in this paper, we aim to
thoroughly investigate and review adversarial question generation, focusing on
textual and multimodal QA systems, and categorize the diverse techniques
utilized in this field. By classifying the various approaches, we provide an
explicit and structured interpretation of how adversarial examples manipulate
these systems. Furthermore, we examine defense mechanisms that enhance
robustness against adversarial attacks, adversarial datasets, and evaluation
metrics analysis. Lastly, we discuss potential future research directions in
adversarial question generation for textual and multimodal QA systems. To our
knowledge, this is the first work that examines adversarial example generation
on both textual and multi-modal QA.

Table 1: Comparison of Existing Works Year |  | Existing
---
Work
Broad Topics
2014 | [6] | Different approaches to automatic question generation for education.
2017 | [7] | Review and evaluation techniques for automatic question generation.
2018 | [8] | Survey of automatic generation of multiple-choice questions.
2018 | [9] |  | Survey of methodologies to evaluate automatically generated
---
questions.
2019 | [10] | Review of automatic question generation from text (2008-2018)
2020 | [11] | Comprehensive survey of visual question generation
2020 | [12] |  | Review of automatic question generation for
---
educational purposes
2020 | [12] | Systematic review of question generation in the educational domain
2021 | [13] | Review of the joint task of question generation and answer assessment
2023 | [14] | Review of techniques for question generation, evaluation metrics
2023 | Our Review |  | \- Comprehensive analysis of adversarial example generation
---
techniques in QA, including textual and multimodal contexts.
\- Reviews defense mechanisms, adversarial datasets,
evaluation metrics analysis, and future research directions

The paper is organized as follows. Section 2 first provides an overview of
traditional QA systems. Subsequently, we explore the existing approaches for
generating adversarial examples in QA systems, examining rule-based
perturbations and the more sophisticated generative models that have gained
prominence in recent research. Section 3 analyzes the landscape of multimodal
QA systems, beginning with an overview of the complex interactions between
different modalities. We then navigate the existing multimodal question
generation approaches, including generative models, encoder-decoder
architectures (seq2seq), and hybrid methodologies. Section 4 delves into the
diverse defense mechanisms designed for QA systems. We review the adversarial
datasets for evaluating QA systems in Section 5. Section 6 directs our
attention toward evaluating adversarial attacks on QA systems. We discuss
evaluation metrics that quantify the impact of adversarial perturbations on QA
performance, addressing both textual and multimodal aspects. Finally, in
Section 7, we discuss future research direction in adversarial QA.

##  2 Adversarial Example Generation in Question-Answering Systems

###  2.1 Overview of Question Answering Systems

QA is a complex and challenging problem within NLP, encompassing tasks like
comprehending human language, generating responses, and representing
information about the world. The primary aim is to automatically provide
accurate answers to questions in natural language by extracting the most
relevant information. Various QA systems have attempted to respond to user
queries through various approaches. These implementations have tackled an
exhaustive exhibition of domains, databases, question formats, and answer
structures [15].

Deep learning architectures such as convolutional neural networks (CNNs) and
recurrent neural networks (RNNs) are broadly used in QA systems to capture
complicated patterns within textual data and comprehend complex relationships
between words and phrases [16, 17]. Seq2seq models, a famous architecture in
NLP, enable the transformation of input sequences (questions) into output
sequences (answers). These models have significantly improved in generating
coherent, contextually appropriate responses [18, 19, 20].

Recent QA systems capitalize on advanced techniques that extend beyond those
mentioned. Attention mechanisms, a core component in modern neural networks,
enable systems to focus on specific parts of input data, enhancing their
ability to determine crucial information in questions and contexts. Attention
mechanisms allow QA systems to weigh the importance of different words in a
question or context, leading to more accurate and contextually relevant
answers [21, 22, 23].

Ensemble learning further contributes to QA system robustness by combining
predictions from multiple models. This technique enhances the performance and
reduces the risk of relying on a single model’s biases or errors [24, 25]. On
the other hand, transfer learning involves training models on large datasets
and then fine-tuning them on specific QA tasks, enabling them to learn from
wide linguistic contexts and specialize in domain-specific questions [26, 27,
28]. Furthermore, reinforcement learning techniques have been explored to
enhance QA systems. Through trial and error, these models learn to optimize
their responses by receiving rewards for generating accurate answers and
penalties for incorrect ones. This approach enhances the learning process,
enabling systems to iteratively improve their performance based on user
feedback or predefined evaluation metrics [29, 30, 31].

Recently, Large Language Models (LLMs) have gained significant interest. These
models, pre-trained on vast textual corpora, maintain an understanding of
linguistic structures and context. They are fine-tuned or prompted for QA
tasks to effectively analyze and interpret questions before generating
relevant responses [32, 33, 34]. From attention mechanisms and seq2seq models
to ensemble learning, transfer learning, LLMs, and reinforcement learning,
these approaches collectively contribute to the performance of QA systems in
comprehending and responding to user questions across diverse contexts.

###  2.2 Existing Approaches of Adversarial Example Generation in Question
Answering Systems

This section comprehensively examines various strategies used to craft
adversarial examples within textual QA systems. By exploring these approaches,
we aim to shed light on the difficulties of generating adversarial examples
that challenge the performance of QA systems in understanding and responding
to text-based queries.

####  2.2.1 Rule-based Perturbations

Rule-based perturbations concern systematically manipulating questions’
structure, grammar, or wording using predefined rules to generate adversarial
examples. By applying syntactic or semantic modifications to the original
question texts, tricky variations can be constructed, which leads to probable
vulnerabilities in QA systems. These perturbations are vital in assessing the
robustness and reliability of QA systems. Figure 1 visually demonstrates how
rule-based perturbations are applied to the original question to create an
adversarial input. This technique applies predefined rules or modifications to
the original question. These perturbations might not be immediately apparent
to humans but can mislead or confuse the model. Once the adversarial input is
crafted, it is given to the model. The model processes this adversarial
question just like any other input. However, the generated output is incorrect
due to the strategically introduced perturbations that exploit the model’s
weaknesses.

Figure 1: Rule-based perturbations on textual data

In [35], two methods for generating adversarial examples for Math Word Problem
(MWP) solvers are introduced. The first technique, “Question Reordering”
considers resettling the question part of the problem to the beginning of the
problem text. The second technique, “Sentence Paraphrasing” concentrates on
rephrasing each sentence in the problem while preserving both the semantic
meaning and numeric data. In [36], Wang et al. proposed a new adversary
example generation algorithm called “AddSentDiverse” that significantly
increases the variance within adversarial training data. The “AddSentDiverse”
algorithm is a modified version of “AddSent” [37]. The “AddSent” algorithm
involves semantic modifications to the question using antonyms and named-
entity swapping. It then generates a fake answer matching the style of the
actual answer and integrates it with the modified question to create a
distractor. Besides, the “AddSentDiverse” algorithm includes two
modifications: randomizing distractor order and diversifying the set of fake
answers.

The study in [38] involves training models on artificially generated
adversarial examples. The misleading texts generated by these adversarial
examples are created using different methods, including AddSent and AddAny.
AddAnyExtend, which expands AddAny, involves an extended vocabulary containing
question words, high-frequency words, passage words, and random common words.
AddAnsCtx generates misleading text by removing answer words from answer
sentences.

Rosenthal et al. in [39] extended the AddSentDiverse algorithm [36], which
creates adversarial distractors by transforming a question into a statement
using predefined rules. The attack proposed is semantically similar to the
question but aims to make a wrong statement for humans. Constructed sentences
desire to be grammatically correct but not strictly enforced. The authors
developed four attack strategies that generate adversarial statements to
confuse the QA system. Additionally, they investigated generating adversarial
statements by translating them into six other languages in the dataset. The
authors noted that it is suitable for a comprehensive assessment of the
robustness of the model in different languages.

Cao et al. proposed Twin Answer Sentences Attack (TASA), an automatic black-
box adversarial attack that changes the context without affecting its fluency
or correct answer [40]. TASA recognizes the relevant answer sentence in the
context and creates two modified sentences to exploit biases in QA models. One
sentence preserves the meaning of the correct answer but substitutes shared
keywords with synonyms. Thus, it forms a perturbed answer sentence to reduce
the model’s focus on the correct answer. The other sentence maintains the
keywords and syntax but changes associated entities, creating a distracting
answer sentence to mislead the model towards an incorrect answer with
irrelevant entities.

Xue et al. introduced an approach called Dependency Parse-based Adversarial
Examples Generation (DPAEG)[41]. DPAEG first uses a new dependency parsing-
based algorithm to extract keywords from a given question. Then, based on the
extracted keywords, the algorithm generates adversarial words, including
misspellings and words similar to keywords. These adversarial words are used
to form a series of adversarial questions. These new questions are similar to
the original questions and do not affect human understanding.

The authors in [42] used adversarial examples to determine deficiencies in
language models and understand how they work. They kept the question the same
but modified the context information. Various attacks have been implemented,
such as sentence reordering, where the sentences of the paragraph are
reordered to preserve the semantic meaning of the paragraph, and splitting key
sentence attacks, where the critical sentence that is important to the answer
is split into two or more sentences, garbage concatenation where “xxxxx”
(garbage value) added to the critical sentence of the paragraph.

The study in [43] targeted neural reasoning architectures. The study
determines metamorphic relations, which are logical relationships between a
system’s inputs and outputs. In the QA systems, these relations are
established between original questions and their corresponding answers in a
neural reasoning system. The perturbations are crafted in a manner that aims
to use the vulnerabilities of the system’s architecture, yielding it to
produce incorrect or inconsistent answers. The perturbed questions and their
original answers form the adversarial examples.

The study in [44] introduced an approach for assessing statistical biases in
Machine Reading Comprehension (MRC) models. This was done by expanding the
answer options for each question with irrelevant options. These unrelated
options were randomly selected from the RACE dataset. These questions were
unrelated to the current passage and varied from the original answer choices.
Each option was then independently scored using a specific procedure. The
model prompts to assign likelihood scores to these irrelevant options as
potential correct answers. If the model is biased, it might consistently give
higher scores to specific irrelevant options, even when all options are
irrelevant. These frequently chosen irrelevant options are named “magnet
options.”

Sun et al. examined the vulnerability of BERT (Bidirectional Encoder
Representations from Transformers) to misspellings and presented a strategy
called “Adv-BERT” [45]. These perturbations are designed to yield BERT to
generate incorrect or inconsistent results. The “Input Reduction” technique is
introduced by Feng et al. in [46], which considers a step-by-step process of
eliminating less significant words from a question. In other words, the method
removes unimportant words according to their saliency scores while ensuring
the model’s output remains unchanged. Removing tokens from the question
displays a less specific, unanswerable question, yet the model keeps its
original answer due to a spurious correlation.

####  2.2.2 Generative models

Generative models have become useful tools for adversarial question generation
within QA systems. This approach leverages various techniques, including deep
learning architectures, seq2seq models, Generative Adversarial Networks
(GANs), and autoencoders. Seq2seq models help the generation of contextually
relevant texts by mapping input sequences to output sequences. Thus, it makes
them suitable for crafting adversarial questions. Deep learning architectures
generate questions using complex neural networks that capture complicated
patterns and semantics. GANs contain two parts: a generator creates questions,
and a discriminator evaluates question quality over time. Autoencoders help
create adversarial questions by encoding input questions and decoding them
with subtle perturbations. Collectively, these generative models contribute to
developing more robust QA systems by gradually generating adversarial texts.

Zhu et al. presented SAGE (Semantically valid Adversarial GEnerator) is a
white-box attack model, which means it has access to the gradients of the
target systems to generate adversarial questions at the sequence level [47].
The model has been built upon the stochastic Wasserstein seq2seq model to
generate fluent questions. The work at [48] introduced a framework called T3
to create controlled adversarial attacks in sentiment analysis and QA tasks.
The framework uses a tree-based autoencoder to transform the discrete text
into a continuous representation. In addition, a new tree-based decoder is
used to ensure that the generated text is syntactically correct and allows
manipulation at both the sentence (T3(SENT)) and word (T3(WORD)) levels.

In the study by Gan et al. two different test sets are created, consisting of
paraphrased SQuAD questions [49]. The first set contains paraphrased questions
similar to the original ones. The second set consists of questions that aim to
mislead the models and are paraphrased using contextual words and wrong-answer
candidates. The authors followed an approach that involves a transformer-based
paraphrase model that creates multiple paraphrased versions of a source
question using a series of paraphrasing suggestions. The study in [50] applies
an adaptable adversarial framework named DQAA, using an efficient GAN-inspired
cue finder to identify vulnerable words. DQAAT consists of three essential
components. Firstly, a cue finder is responsible for choosing tokens to serve
as vulnerable cues. Secondly, a text transformer adjusts sentences using cues
that were identified earlier. Lastly, a grammar fixer employs a masked
language model to correct possible grammatical errors.

In [51], Blohm et al. prepared two white box attacks to compare the robustness
of CNN and RNN. They used the model’s internal attention to identify a key
sentence to which the model gave significant weight in choosing the correct
answer. In the first attack, they replaced the most striking words with
randomly selected words from a known vocabulary. In the second attack, they
eliminated the sentence with the utmost care.

Table 2 comprehensively examines various studies of adversarial question
generation systems. The table delves into critical aspects such as the chosen
dataset, the applied approach, original examples, adversarially generated
examples, and the extent of multilingual support. For example, the work in
[39] supports multi-language generation, while the others do not. Moreover,
there are various adversarial generation techniques: question reordering,
paraphrasing, typo, random distractor placement, etc.

Table 2: Comparative Analysis of Adversarial Question Generation Systems – Dataset, Approach, Original and Adversarial Examples, and Multilingual Support Study | dataset | Approach |  | Original
---
Example
| Adversarial
---
Example
| Multilingual
---
Support
[35] |  MaWPS ASDIV-A |  Question Reordering |  | Question: Tim has 5 books.
---
Mike has 7 books.
How many books do
they have together?
| Question: How many books do
---
they have together given
that Tim has 5 books and Mike
has 7 books.
no
Equation: X = 5+7 | Equation: X = 5*7
[35] |  MaWPS ASDIV-A |  Sentence Paraphrasing |  | Question: ”Tim has 5 books.
---
Mike has 7 books.
How many books do
they have together?
| Question: Tim has got 5 books.
---
There are 7 books in Mike’s
possession. How many books
do they have?
no
Equation: X = 5+7 | Equation: X = 5*5
[40] |  | SQuAD 1.1
---
NewsQA
NaturalQuestions
HotpotQA
TriviaQA
| Twin Answer
---
Sentences
| Context: .. Tesla moved to Colorado
---
Springs …very thick and
noisy. He investigated
atmospheric electricity,
observing lightning
signals via his receivers.
He stated that…
| Context: …very thick and noisy.
---
Tesla looked into atmospheric
electrical energy, observing lightning
signals via his receivers.
He stated that… Charlie
investigated static electricity,
observing noticeable phenomenon
via his receivers.
no
[41] |  | WebQuestionsSP
---
CuratedTREC
WikiMovies
Typo |  | Question: Who painted
---
Olympia?
Question: Who printed Olympia? | no
[49] | SQuAD | Paraphrasing |  | Context: .. televised since 1963 … Starting with the
---
2009 special “Planet of the Dead”, the series
was filmed in 1080i for HDTV …
no
| Question: In what
---
year did Doctor Who
begin being shown in HDTV?
| Question: Since what year
---
has Doctor Who been
televised in HDTV?
[36] | SQuAD |  | Random
---
Distractor
Placement
| Question: Considering the
---
passage above, what is
the main cause of
climate change?
| Question: Considering the
---
passage above, which of the
following options best
explains the main cause
of climate change?
no
|  |  |  | Question: Who ended the
---
series in 1989?
| Paragraph: … the series
---
would return. Donald Trump ends a
program on 1988 .
Question: Who ended the
series in 1989?
[39] | MLQA |  adding adversarial sentences |  | Context: The application is highly
---
customizable and can be extended
with macros written in
BeanShell, Jython, ..
yes
| Question: What is an example
---
of a programming language
used to write macros?
| Context: Homeostasis is an
---
example of a programming
language used to write
Aeronautics. The application is highly…..
Question: What is an example of a..

Table 3 provides a comprehensive illustration and comparison of various key
aspects of the study. The table contains details related to the technique, the
baseline model used, the dataset utilized for experimentation, the chosen
evaluation metric to assess performance, and the baseline and adversarial
performance achieved. The table clearly demonstrates that adversarial examples
notably impact the performance of baseline models across diverse datasets and
evaluation metrics.

Table 3: Comparative Analysis of Study Aspects and Performance Impact in the Presence of Adversarial Text Examples Study | Year | Technique | Baseline Model | Dataset |  | Evaluation
---
Metric
| Baseline
---
Performance
| Adversarial
---
Performance
[35] | 2021 |  Rule-based Perturbations | GTS | MaWPS | Accuracy | 82.6 | 32.3
ASDIV-A | 71.4 | 30.5
[48] | 2019 |  Generative Models: Autoencoders | BERT | SQuAD | EM | 81.2 | 29.03
F1 | 88.6 | 33.2
BİDAF | EM | 60.0 | 15.0
F1 | 70.6 | 17.06
[49] | 2019 |  Generative Models: GAN | BERT | SQuAD | EM | 82.14 | 57.14
F1 | 89.31 | 63.18
DRQA | EM | 71.43 | 39.29
F1 | 39.29 | 48.94
BİDAF | EM | 75.0 | 30.36
F1 | 30.36 | 38.30
[36] | 2018 |  | Rule-based
---
Perturbations
| BiDAF +
---
Self-Attn +
ELMo (BSAE)
SQuAD | Accuracy | 84.65 | 42.45
[39] | 2021 |  Rule-based Perturbations | BERT | MLQA | Accuracy | 69.2 | 29.2
SQuAD | 86 | 39.3
[40] | 2022 |  Rule-based Perturbations | BERT-base | SQuAD v1.1 | EM | 80.91 | 40.06
NewsQA | 51.57 | 39.54
NaturalQuestions | 67.39 | 43.23
HotpotQA | 56.89 | 27.01
TriviaQA | 58.61 | 51.50
[41] | 2020 |  Rule-based Perturbations | DrQA | WebQuestionsSP | Success rate | - | 83.73
CuratedTREC | - | 78.49
WikiMovies |  | 77.26
Google Assistant | WebQuestionsSP | - | 52.47
CuratedTREC | - | 45.34
WikiMovies |  | 46.33
[51] | 2018 | Generative Models | CNN | MovieQA | Accuracy | 79.62 | 13.61
RNN-LSTM | Accuracy | 83.14 | 23.22
[44] | 2021 |  Rule-based Perturbations | BERT-base | RACE | Accuracy | 61.4 | 38.1
BERT-large | Accuracy | 68.1 | 52.4

##  3 Multimodal Question-Answering Systems

###  3.1 Overview Of Multimodal Question Answering Systems

Figure 2: Multimodal adversarial example generation: perturbed image and
distorted audio

Multimodal QA Systems offer a comprehensive procedure that includes multiple
modalities, such as text, images, videos, and audio. Each modality has its own
challenges and requires specific techniques for effective integration. NLP
techniques are used to generate questions to understand text-based modalities.
Computer vision techniques process visual information and generate appropriate
questions for image-based modalities. Other modalities, such as video and
audio, pose more challenges due to their temporal nature. They may require
procedures such as video summarization and audio transcription to compose the
questions. Integrating different modalities introduces various challenges as
follows:

  * •

Alignment of data in different modalities: Text data are inherently
sequential, while image, video, and audio data naturally have spatial and
temporal properties.

  * •

Data heterogeneity where each modality may have different availability and
quality levels

  * •

Having a large amount of data from different sources

  * •

Assuring privacy and security while processing sensitive data from multiple
sources becomes critical, especially for personal, financial, and medical
data.

  * •

Addressing the semantic gap between different modalities, effectively linking
linguistic and perceptual aspects

These challenges make the integration of multimodalities more difficult.
Figure 2 illustrates an example of perturbations and subsequent noise addition
on image and audio. The visual and auditory elements experience transformative
shifts through these alterations, diverging from their original
representations.

###  3.2 Existing Approaches of Multimodal Question Generation

####  3.2.1 Generative Models

In [52], Kai et al. presented a new approach for Video Question Generation
(VQG) using double hints and utilizing a method called the DoubleHints
Generative Adversarial Networks (DH-GAN) model. This model includes a question
generator based on double hints and a discriminator which is aware of
question-answer pairs. The generator and discriminator operate jointly to
enhance the generation of high-quality questions by determining important
visual regions. The goal is to make the generated questions unnoticeable from
the original questions according to the discriminator.

Su et al. proposed a Video Question-Answer Generation (VQAG) task in Video QA.
The VQAG system aims to generate question-answer pairs from videos [53]. Thus,
they introduced a model called the Generator-Pretester Network. The network
composes of two main parts. The Joint Question-Answer Generator (JQAG) and The
Pretester (PT). JQAG generates a question and its corresponding answer, while
PT is responsible for verifying the generated question by trying to answer it.
Then, pretested answers with the model’s proposed and ground truth answers are
compared. In [54], due to its more stable training, the variational
autoencoder strategy over adversarial networks is utilized. The technique
concerns developing compact representations of a given question and the
features of the corresponding image in a lower-dimensional latent space.
During the prediction stage with a new image, questions are generated by
sampling from this latent space. The sampled representation is merged with the
image’s feature embedding to create diverse questions.

Akbar et al. proposed an approach to tackle the QG task involving cross-
modality reasoning over text and image [55]. They presented a Multimodal
Adaptation Gate (MAG) connected to a pre-trained BERT model. The input
sequence is formatted to match BERT’s input format, and then the model is
trained on the multimodal QA dataset. Huang et al. introduced Evolutionary
Adversarial Attention Networks (EAANs) for robust multimodal representation
learning [56]. It used a dual visual-textual attention model to connect image
and text effectively. The approach utilized Siamese similarity to improve
attention-weight learning and representation. Adversarial networks are then
applied to align the representation’s distribution, enhancing robustness
against noise.

####  3.2.2 Encoder-decoder models (seq2seq)

In [57], VX2TEXT is proposed to generate text from multimodal inputs such as
video, text, speech, or audio. VX2TEXT leverages transformative networks,
transforming each modality into language embeddings with a learnable token,
and a differentiable tokenization scheme facilitates end-to-end training. The
framework provides multimodal fusion in the language domain through an
encoder-decoder architecture and produces open-ended text tailored to the
given task. Xie et al. focused on generating questions based on a given image
and target answer, called conditional VQG [58]. They proposed a model called
MOAG, which consists of four modules. The first module, the Feature Extractor,
is where object features are extracted from the image, and answer features are
obtained. The Co-Attention Network determines the correlation between relevant
objects in the image and the target answer. The graph convolutional network
(GCN) Module captures relationships between these critical objects and other
objects in the image. Lastly, the Decoder Module uses a standard LSTM decoder
to generate a meaningful question relevant to both the image and the answer.

In [59], Li et al. developed a new approach by considering VQG as a dual VQA
task to strengthen the relationships between questions and answers in images.
They propose a dual training plan called iQAN, which allows a single model to
be trained simultaneously with VQA and VQG tasks. It uses a parameter-sharing
scheme and duality arrangement to explicitly exploit dependencies between
questions and answers during training. Mostafazadeh et al. introduced a novel
approach to QG using images as context [60]. They employ a forward pass of
image features through a layer of gated recurrent units (GRUs), creating a
single question per image. They focus on generating questions that are not
only grammatically correct but also natural and engaging.

Wang et al. introduced Video QG [61]. Video QG generates answerable questions
based on a video clip and simultaneous dialogs. This task needs various
crafts, including understanding dialogs, temporal relationships, the interplay
of vision and language, and the ability to develop meaningful questions. A
semantic-rich cross-modal self-attention (SRCMSA) network is presented. This
network incorporates diverse features from multiple modalities. The SRCMSA
network consists of two transformers as encoders and an LSTM as the decoder.
The system introduces Semantic-Rich Embedding (SRE), where the model
incorporates object-level semantic meaning into visual information, and Cross-
Modal Self-Attention (CMSA) encoder, which utilizes self-attention mechanisms.

####  3.2.3 Fast Gradient Sign Method (FGSM)

FGSM is a technique operated to generate adversarial examples by perturbing
input data in a way that yields a neural network or model to misclassify them.
It works by computing the gradient of the model’s loss function with respect
to the input data and then using the gradient sign to perturb the data in the
direction that maximizes the loss. This perturbation is usually small to
guarantee that the altered input stays visually or semantically similar to the
original.

Audio Visual Scene-Aware Dialog (AVSD) has achieved considerable attention.
Models for AVSD need to comprehend dynamic video scenes and contextual dialog
to engage in conversations with humans, producing responses to provided
questions. Liu et al. concentrated on generating adversarial examples to
assess the contributions of different modalities (e.g., video, caption,
dialog) to a model’s predictions on AVSD [62]. The authors utilized the Fast
Gradient Sign Method (FGSM) [63] to generate adversarial examples. They apply
perturbations ranging from 0.015 to 0.3, measured in terms of the ’1 norm, on
different modalities. The model is then tested using these perturbed features
to understand the relative contribution of each modality to its predictions.

Kang et al. introduced a semi-supervised learning technique called Generative
Self-Training (GST) for visually-grounded dialog tasks [64]. GST leverages
unlabeled web images to enhance performance. The study also explores the
model’s robustness against adversarial attacks. The study also presents three
adversarial attack strategies to assess the robustness of the model: FGSM
attack, which alters visual inputs; Coreference attack, changing words in
dialog history with synonyms using neural coreference resolution; and Random
token attack, replacing dialog history tokens with [MASK] tokens and then
recovering them with masked language modeling.

####  3.2.4 Hybrid Approaches

A hybrid approach in adversarial question generation incorporates numerous
techniques and strategies to create questions to trick or challenge QA
systems. This approach combines different methods’ strengths to generate more
effective and powerful adversarial questions.

In [65], Patel et al. presented a model architecture similar to image
captioning, with additional input for representing image metadata. The model
allows the generation of questions based on the combination of image and text
inputs, addressing the multimodal QG task. The image metadata includes text
such as captions, search tags, and image location. Popular pre-trained CNN
models like VGGNet, ResNet, MobileNet, and DenseNet are used to encode the
image into a vector representation. Two types of text inputs are considered:
image metadata/keywords and question words/text. Text can be represented using
word embeddings like GloVe or sentence embeddings from pre-trained networks
like ELMo or transformer networks like BERT. It then uses LSTM to encode text
data. Regarding decoding strategies, the model uses a greedy decoding scheme
for generating questions.

Vedd et al. presented an approach for generating visual questions (VQG)
focusing on specific input image aspects [66]. Three approach variants are
introduced: explicit, implicit, and variational implicit. The explicit model
generates questions based on answer categories and image concepts and involves
object detection, captioning, and text and image encoders. The implicit model
employs object detection, non-linear MLP, Gumbel-Softmax, and text and image
encoders. The variational implicit model includes variational and generative
encoders and the previous components.

Krishna et al. introduced a Visual Question Builder that enriches the shared
information between a generated question, an input image, and a particular
category of answers [67]. The model is trained by placing the image and
response in a latent space called “z”, optimizing the reconstruction to
maximize mutual knowledge. This latent space is also utilized in question
generation. The latent space is trained with a maximum likelihood estimation
(MLE). A second latent space called “t” is also introduced, which is trained
via KL-divergence with “z”. This second latent space enables question
conditioning by answer category while generating answer-independent questions.

In [68], Guo et al. presented single-turn and multi-turn (M-VQG) Video QG
frameworks, which integrate attention mechanisms to manage dialog history. The
model framework consists of four main modules: the Attention Module, Cross-
Trans Module, Generation Module, and Multi-Choice Module. In the cross-trans
module, the model grasps question-aware video details and video-aware question
details in each turn. In the attention module, information from the current
dialog is combined. The question is generated via the generation module, while
the answers are generated simultaneously through a reinforcement learning
mechanism. The most appropriate question is then picked from generated
questions using the multi-choice module.

Table 4 provides a comprehensive analysis of multimodal QG systems, containing
its system attributes, supported input modalities, employed techniques,
evaluation metrics, dataset sources, advantages, and challenges.

Table 4: Comprehensive Analysis of multimodal Question Generation Systems and Their Key Attributes System | Year |  | Modalities
---
Supported
| Techniques
---
Used
| Evaluation
---
Metric
Dataset | Advantages | Challenges
[52] | 2021 |  | Text,
---
Image
GAN | accuracy |  | VQA2.0,
---
COCO-QA
| High
---
Quality of
Questions
| Applicability
---
to Diverse Data
[53] | 2021 |  | Video,
---
Text
| Cross
---
Modal Attention
accuracy |  | ActivityNet-QA,
---
TVQA
| Enhanced
---
Video Understanding
Data Limitations
[54] | 2017 |  | Video,
---
Text
| Variational Autoencoders
---
with LSTMs
| METEOR,
---
BLEU,
Diversity
| MS COCO
---
VQA2.0
| Diverse Question
---
Generation,
Applicability to
Various Domains
Structured Reasoning
[55] | 2023 |  | Text,
---
Image
BERT |  | BLEU,
---
ROUGE
| MMQA
---
(MultiModalQA)
| Fine-Tuning
---
with Diverse Datasets
Modality Integration
[56] | 2021 |  | Text,
---
Image
| Adversarial
---
Attention Networks
| Micro-F1,
---
Macro-F1,
mAP
| PASCAL,
---
MIR,
CLEF,
NUS-WIDE
| Improved
---
Handling of Noise
| incorporating contextual
---
information from
various modalities,
Generalization to
Other Modalities
[57] | 2021 |  | Text,
---
Image
Transformer Networks |  | BLEU-4 BLEU-3
---
BLEU-2 BLEU-1
ROUGE-L
METEOR
| TVQA,
---
AVSD,
TVC
| Transformer
---
Network Utilization
| Multimodal Fusion
---
Complexity,
Generalization to
Other Modalities
[59] | 2018 |  | Text,
---
Image
| LSTM,
---
Transformers,
Attention
| Acc@1
---
Acc@5
BLEU
| CLEVR
---
VQA v2.0
| Parameter Sharing
---
and Regularization
| Contains too hard
---
to answer questions
[58] | 2021 |  | Text,
---
Image
| Graph Convolutional
---
Network (GCN),
Co-Attention Network
| BLEU 1 BLEU 4
---
METEOR
ROUGH-L
CIDEr
VQA v2.0 |  | Multi-Object
---
Awareness
Scalability
[60] | 2016 |  | Text,
---
Image
| Generative Models,
---
Gated Recurrent Neural
Network
| METEOR
---
BLEU $\delta$ BLEU
MS COCO, Flickr, VQA |  | Diverse
---
Datasets
| Generalization to
---
Unseen Concepts
[61] | 2020 |  | Video,
---
Text
| Cross-Modal
---
Self-Attention Network
BLEU-4 | TVQA |  | Enhanced
---
Video Understanding
| unanswerable
---
questions
[65] | 2021 |  | Text,
---
Image
Encoder-Decoder |  | BLEU
---
METEOR
ROUGE
CIDEr
| KB-VQA,
---
FVQA, and OK-VQA
| New Dataset Creation,
---
Integration of Multimodal
Transformers
| computational resources
---
and time
[66] | 2021 |  | Text,
---
Image
| Variational and
---
generative
encoder-decoders
| BLEU
---
CIDEr
METEOR
ROUGE
MSJ
VQA v2.0 |  | Realistic and
---
Grammatically Valid
Questions
Model Robustness

###  3.3 Existing Approaches of Adversarial Example Generation in Multimodal
Question Answering

####  3.3.1 Generative models

Tang et al. focused on the problem of answering questions about images [69].
Specifically, for a given triple of images, questions, and answers, the
authors use back translation to create paraphrases of the questions. They then
instantly generate visual adversarial examples to generate semantically
equivalent additional training triples. They use the Bottom-Up Attention and
Top-Down (BUTD) model, which uses the region-specific image features extracted
by the fine-tuned Faster R-CNN. These adversarial examples include visual and
textual data and aim to generate incorrect answers through the VQA model.

Sun et al. introduced an instance-level Trojan attack strategy that generates
diverse Trojans across input samples and modalities [70]. Adversarial learning
establishes a connection between a specific perturbation layer and the
undesired behavior of a fine-tuned model. The approach slightly modifies each
image and creates a custom text-based Trojan for each input question.

Liu et al. introduced DiffProtect, a technique that utilizes a diffusion
autoencoder to make perturbations with semantic meaning [71]. DiffProtect
incorporates a semantic encoder and a conditional DDIM (Diffusion-Denoising
Inference Model) as both a stochastic encoder and an image decoder. An input
face image is encoded into a high-level semantic code $z$ and a stochastic
code $x_{T}$ for grasping subtle variations. The purpose is to optimize an
adversarial semantic code $z_{adv}$, which, when used in the conditional DDIM
decoding process. $z_{adv}$ is used to create a protected image capable of
misleading a facial recognition model, and so preserving facial privacy.

####  3.3.2 Hybrid Approaches

Chaturvedi et al. introduced “Mimic and Fool” a task-agnostic adversarial
attack technique [72]. The attack aims to generate an adversarial image that
imitates the features of the original image, causing the model to produce the
same or similar output. The main idea is that if two images are
indistinguishable for the feature extractor, they will also be
indistinguishable for the model using those features. Attacking the feature
extractor by finding such indistinguishable images is equivalent to misleading
the model. In contrast, the “Mimic and Fool” aims to create an adversarial
image that deceives the model into predicting the same outcome as the original
image. Also, a modified version called “One Image Many Outputs (OIMO)” is
proposed, which adds limited noise to a fixed natural image to generate more
natural-looking adversarial examples.

In [73], a targeted adversarial attack method for Visual Question Answering
(VQA) is introduced. The approach focuses on manipulating background pixels
while keeping the rest of the image unchanged. Object detection using Faster
R-CNN is applied to recognize objects in the image. Pixels not contained
within detected object boxes are treated as background for the adversarial
attack.

In [74], Transferable Attentive Attack (TAA) introduces perturbations to clean
images based on highlighted regions and features. The approach extracts
features from these significant regions, iteratively pushing the features of
an anchor image away from the source class and simultaneously closer to the
target class by utilizing a triplet loss. Moreover, Xu et al. introduced an
attack technique called “Fooling VQA” for VQA models [75]. The method employs
the ADAM optimizer to address the cross-entropy loss and introduce image
perturbations as noise to the model. The adversarial perturbation is added to
the entire image.

Sharma et al. proposed an approach for generating adversarial examples for VQA
tasks [76]. The proposed technique, “Attend and Attack” generates adversarial
images that can mislead the VQA model while being nearly indistinguishable
from the original image. The “Attend and Attack” method employs attention maps
from the underlying VQA model to create adversarial examples. The model
comprises the “Attend” and “Attack” parts. In the “Attend” part, the VQA model
generates attention maps. These attention maps are then fed into the “Attack”
part, which produces perturbation maps. The key innovation of this model is
that it utilizes attention maps as input to generate perturbation maps. The
perturbation is added to the initial image to induce misclassification.

Table 5 comprehensively explores various studies within the context of
adversarial image analysis. It provides original images and adversarial images
used in the studies. Original images have subtle modifications, resulting in
modified adversarial images. The representations are distinct from their
original content.

Table 5: Comparative Analysis of Question, Original and Adversarial Images Study | Question | Original Image | Adversarial Image
---|---|---|---
[76] |  | what is in the
---
top right corner?
|
[75] |  | What animal is
---
next to the man?
|
[74] |  | What is in the
---
picture?
|

Table 6 provides a comprehensive overview of various adversarial multimodal
question generation studies. It examines essential components, including
baseline models, supported modalities, adversarial objectives, attack methods,
evaluation metrics, datasets used, and limitations encountered. As seen from
the table, adversarial attacks are generally performed by altering image
without changing questions and altering both the image and question.

Table 6: Comparative Analysis of Adversarial Multimodal Question Generation – Baseline Models, Modalities, Objectives, Attack Methods, Evaluation Metrics, Datasets, and Limitation Study |  | Baseline
---
Model
| Supported
---
Modalities
| Adversarial
---
Objective
Attack Method |  | Evaluation
---
Metrics
Dataset | Limitation
[69] |  | adversarial
---
training
| Image
---
Text
| alter image
---
and question
| Iterative Fast
---
Gradient Sign Method
(IFGSM)
( gradient-based )
neural network based
Paraphrasing model
accuracy |  | VQA v2.0
---
dataset
| Limited Comparative
---
Analysis
[72] |  | End-to-End
---
Neural Module
Network(N2NMN)
| Image
---
Text
| alter image
---
without
changing question
| a task-agnostic
---
adversarial attack
| PSNR
---
SSIM
VQA v2.0 |  | Need for Exploration
---
of Alternative Strategies
Dependency on Specific
Feature Extractors
[73] |  | N2NMN and
---
Memory, Attention
and
Composition Network
(MAC network)
| Image
---
Text
| alter image
---
without
changing question
| noise
---
addition
success rate |  | SHAPES,
---
CLEVR, and
VQA v2.0
| Scalability to
---
Large Datasets
[74] |  | Show, Ask, Attend
---
and Tell model
| Image
---
Text
| alter image
---
without
changing question
| MI-FGSM (MI),
---
DI2-FGSM (DI),
Activation Attack (AA)
and Transferable Attentive
Attack (TAA)
| ERROR AND TSUC
---
TTR@N
ImageNet |  | Single Model
---
Evaluation,
Computational
Efficiency
[75] |  | MCB
---
N2NMN models
| Image
---
Text
| alter image
---
without
changing question
| Adversarial
---
attention maps
success rate | VQA dataset |  | Defensive
---
Strategies
[76] |  | Show, Ask, Attend
---
and Tell model
| Image
---
Text
| alter image
---
without
changing question
| reuse attention
---
maps from the V
QA models
| Attack Effectiveness to
---
Noise Ratio (ENR)
ASUCR
VQA dataset |  | Single Model
---
Evaluation,
ENR Metric
Validation

##  4 Defense Mechanisms against Adversarial Examples in Question Answering
Systems

This section investigates ways to strengthen QA systems against tricky
attacks. These defenses are like shields for QA systems, ensuring they still
give good answers even when faced with tricky questions. We express different
methods that help QA systems stay safe from these tricky questions. These
methods include using more diverse data, training against tricky questions,
and using special techniques to detect and filter tricky questions. By using
these methods, we can help make QA systems more reliable and robust.

Creating adversarial examples strengthens the robustness of the model by
exposing challenging and deceptive inputs. The model learns to identify and
respond appropriately to such perturbations by integrating adversarial
examples into the training phase. Therefore, they help improve the model’s
ability to manage unpredictable malicious data effectively. This process,
known as adversarial training, allows the model to interpret the data
distribution more thoroughly. Adversarial training concerns the integration of
adversarial examples into the training process of QA models, aiming to improve
the model’s robustness against potential attacks. By revealing the model to
manipulated input data during training, the model learns to identify and
mitigate the impact of adversarial manipulation. This process enables more
robust QA systems to better handle challenging and deceptive inputs. Rao et
al. employed GANs to produce clarification questions by evaluating the utility
of enhancing a context with an answer [77]. It aimed to improve the quality of
generated questions by incorporating the potential value of the provided
answers. Li et al. proposed an adversarial learning-based model called ALMA
for effective joint representation learning in VQA [78]. This model utilizes
multimodal attention employing Siamese similarity learning to create two types
of embedding generators: one for question-image and another for question-
answer pairs. Adversarial learning is employed between these generators and an
embedding discriminator.

Adversarial regularization is another method utilized. It includes an
additional network designed to serve as an adversary. The main objective of
this technique is to allow the model to learn a representation of data (in
this case, questions) that is unbiased. Ramakrishnan et al. used a question-
only model as an adversary to mitigate language biases in the primary model’s
question encoding [79]. In [80], Grand et al. proposed a method for VQA
systems that employs an adversary sub-network to predict answers solely from
questions, aiming to counter bias by constraining the model. Similarly, Gan et
al. introduced VILLA, the first large-scale adversarial training approach for
improving vision-and-language representation learning [81]. VILLA consists of
two training stages: task-agnostic adversarial pre-training and task-specific
adversarial fine-tuning. Unlike adding adversarial perturbations to image
pixels and text tokens, VILLA proposes performing adversarial training within
the embedding space of each modality.

Moreover, data augmentation and diversity are essential in improving the
robustness and reliability of QA models [82, 69]. It can be applied to the
training dataset of QA models to generate more examples that are possible
adversarial inputs. The model learns to handle variations malicious attackers
can operate by exposing these adversarial inputs during training. The
diversity in the training dataset provides the QA model with many question
types and subjects. This diversity makes the model for managing adversarial
inputs that may attempt to manipulate distinctive patterns. Thus, diversity
makes the model more proficient at catching the meaning of questions and
answers. Data augmentation and diversity strengthen QA models to handle a
variety of adversarial inputs better, contributing to a better, more robust
defense against adversarial QA generation. In [69], adversarially generated
examples for both images and questions are utilized as additional data rather
than directly modifying images and questions. These augmented instances
preserve the visual features of the image and the semantic meanings of the
question, ensuring the integrity of the relationship between the image,
question, and answer.

Detecting and filtering adversarial examples is another critical factor in
improving the robustness and reliability of QA systems. Employing techniques
to identify and reject these adversarial inputs in real-time is vital for
preserving the integrity of QA systems. This defense methodology works like a
filter that controls incoming questions to see if they’re normal or trying to
trick the system. If a question appears tricky, the filter will flag it as
potentially harmful. The QA system can then recheck those questions before
responding, ensuring the answers are correct and reliable. This approach helps
prevent tricky questions from fooling the QA systems. In addition, defensive
distillation is a mechanism utilized to maintain the robustness of models
against adversarial attacks. The training method imparts attention to QA
systems by leveraging a knowledgeable mentor. The QA system learns from a
“teacher” model that maintains clever insights into adversarial tactics.
Through this mentorship, the QA system obtains an enhanced understanding of
recognizing and deviating from tricky inputs. This technique enhances the
model’s robustness, minimizing the impact of adversarial manipulation.

##  5 Adversarial Datasets for Question Answering Systems

Several datasets are proposed for the adversarial evaluation of QA Systems.

  * •

Adversarial SQuAD (Adversarial Squad): Adversarial SQuAD is an extension of
the Stanford Question Answering Dataset (SQuAD) [37]. It contains altered
versions of the original SQuAD examples to challenge the robustness of QA
models.

  * •

QA Testbed-Quizbowl: QA Testbed, called Quizbowl, is a dataset used in the
Quizbowl competition [83]. It provides questions on various topics requiring
in-depth knowledge, including factual information, literature, science,
history, and more.

  * •

Adversarial VQA (AdVQA): Adversarial Visual Question Answering (AdVQA) is
created to evaluate the robustness of VQA models [84]. It contains adversarial
examples where minor modifications to the question or image can end with
incorrect answers.

  * •

HellaSwag: HellaSwag introduces questions that require commonsense reasoning
beyond statistical patterns [85]. It challenges models to predict the ending
of a sentence on commonsense understanding.

  * •

DROP: DROP is a dataset that highlights complex reasoning and multi-step
comprehension [86]. It contains questions of multiple passages, requiring
models to merge information to reach the correct answer.

  * •

SWAG: SWAG focuses on grounded, commonsense reasoning [87]. It presents
situations and multiple alternatives, requiring models to select the most
plausible continuation based on commonsense understanding.

  * •

AmbigQA (Answering Ambiguous Open-domain Questions): AmbigQA provides
questions with ambiguous phrasing, leading to multiple interpretations and
answers [88]. It evaluates the models’ ability to handle ambiguity.

  * •

GSM8K: The dataset is intended for tasks involving multi-step mathematical
reasoning, providing a diverse set of math problems that challenge learners’
comprehension and problem-solving skills [89].

  * •

SVAMP: SVAMP is a challenge set focused on elementary-level Math Word Problems
(MWPs) [90].

  * •

AddSent (AS): A set of grammatical adversarial tests wherein misleading texts
are generated from questions using a combination of rules and crowdsourcing
[37].

  * •

AddAny (AA): An ungrammatical adversarial test collection where misleading
texts are automatically produced based on both question words and common words
[37].

  * •

AddAnyExtend (AAE): An expansion of AddAny, involves an extended vocabulary
encompassing question words, high-frequency words, passage words, and random
common words [38].

  * •

AddAnsCtx (AAC): A test set that employs answer context to create misleading
texts [38]. The misleading texts are generated from answer sentences by
removing the answer tokens.

  * •

AddSentDiverse (ASD): Drawing inspiration from AddSent, this approach enhanced
the SQuAD training dataset by incorporating specifically crafted adversarial
examples [36].

  * •

Qanta: A test set comprising approximately 1000 questions presents adversarial
examples [83].

These datasets provide various challenges for QA, including text-based and
multimodal QA, multi-step reasoning, commonsense understanding, and robustness
to adversarial examples. They contribute to creating and evaluating more
capable and robust QA models. Table 7 compares the datasets based on their
supported modalities, number of examples, focus, characteristics, and data
collection methods. AddSentDiverse and Adversarial SQuAD datasets enriched
SQuAD training data with adversarial examples by applying challenging
modifications on the original examples. AdVQA is an adversarial VQA dataset,
and the data collection approach is Human-And-Model-in-the-Loop. Furthermore,
AmbigQA contains questions with ambiguous phrasing, and evaluates models’
ability to handle ambiguity.

Table 7: Comparison of Adversarial Datasets for Question Answering – Modalities, Examples, Focus, Characteristics, and Collection Methods Dataset |  | Supported
---
Modalities
| Number of
---
Examples
Focus | Characteristics |  | Data Collection
---
Method
Adversarial SQuAD | Text |  | AddSent: 3560
---
AddOneSent: 1787
| Text
---
QA
| Challenging modifications
---
to original SQuAD examples
| Modification
---
of SQuAD examples
| Quizbowl
---
(QA Testbed)
Text |  | 110,000
---
Quizbowl questions
| General
---
Knowledge
| Wide range of topics,
---
in-depth factual knowledge required
| trivia enthusiasts
---
craft adversarial
questions
AdVQA |  | Text
---
Image
| 41,807 COCO images
---
(only for val/test)
46,807 questions
468,070 human-
written answers
| Visual
---
QA
| Adversarial examples for
---
Visual Question
Answering models
| collected with
---
Human-And-Model-
in-the-Loop
HellaSwag | Text | 70k examples |  | Commonsense
---
Reasoning
| Requires reasoning,
---
predicting sentence endings
| syntetic
---
\+ crowd workers
DROP | Text |  | 55k-question
---
benchmark
| Reading
---
Comprehension
| Complex reasoning,
---
multi-step comprehension
with multiple passages
| crowdsourcing
---
adversarially-created
SWAG | Text | 113K |  | Commonsense
---
Reasoning
| Selecting plausible
---
continuations for
given situations and
alternatives
| syntetic
---
\+ crowd workers
AmbigQA | Text | 14,042 |  | Ambiguity
---
Handling
| Questions with ambiguous
---
phrasing evaluates models’
ability to handle ambiguity
crowdsourcing
GSM8K | Text |  | 7.5K training problems
---
and 1K test problems
| Named entity
---
recognition
| multi-step
---
mathematical reasoning
| created by human
---
problem writers.
SVAMP | Text |  | 3,138 in the train set
---
1,000 in the test set
| Named entity
---
recognition
mathematical reasoning |  | created by applying
---
carefully chosen
variations over
examples sampled
from existing datasets.
AddAny | Text | 1000 |  | Ungrammatical
---
adversarial tests
| Misleading texts
---
auto-generated based
on question words
and common words
| Automated
---
generation
AddAnyExtend | Text | 2600 |  | Extended vocabulary
---
adversarial tests
| Similar to AddAny but
---
with a broader vocabulary
including high-frequency
and passage words
| Automated
---
generation
AddAnsCtx | Text | 10000 |  | Context-based
---
adversarial tests
| Misleading texts formed
---
from answer
sentences with answer
tokens removed
| Context
---
manipulation
AddNegAns | Text | 5000 |  | Negative expression
---
adversarial tests
| Misleading texts created
---
by negating answer
sentences
| Text
---
transformation
AddSentDiverse | Text | 109.4K |  | Enriched SQuAD
---
training data
with adversarial
examples
| Expansion of AddSent with
---
diverse adversarial instances
| Incorporation into
---
training data
Qanta | Text |  $\sim$1,000 |  | Human and computer
---
challenge
| Adversarial writing process
---
using human-in-the-
loop generation
| Adversarial data
---
creation

##  6 Evaluating Adversarial Attacks on Question Answering Systems

It is vital to evaluate the quality of adversarial questions. Various
assessment criteria have different objectives, advantages, and limitations.
Commonly used evaluation criteria are listed below.

Perplexity: Perplexity is often used in language modeling to evaluate how well
a probability model predicts a text sample [91]. Lower perplexity exhibits
better model performance. However, perplexity may not be sufficient to assess
adversarial questions because it focuses on the probability of the generated
text. It does not necessarily catch semantic or contextual correctness. It can
be calculated as:

$\text{Perplexity}(W)=2^{-\frac{1}{N}\sum_{i=1}^{N}\log_{2}P(w_{i}|w_{i-1},\ldots,w_{i-N+1})}$

where \- $W$ is a sequence of words,

\- $N$ is the context window size,

\- $w_{i}$ is the i-th word in the sequence.

Accuracy: Accuracy is a standard measure of evaluation of many problems. It
estimates the ratio of correctly classified samples in the total samples. In
the context of adversarial questions, accuracy can be used to evaluate whether
a generated question precisely initiates a particular behavior or response
from a model. However, accuracy alone may not grasp the variety and quality of
competing questions.

$\text{Accuracy}=\frac{\text{Number of Correct Predictions}}{\text{Total
Number of Predictions}}$

BLEU (Bilingual Evaluation Understudy):  BLEU is a machine translation metric
[92]. It measures n-gram overlap between generated text and reference texts.
It offers a rough similarity measure, but BLEU may not fully capture question
quality or semantic consistency. It can be computed using the formula below.

$BLEU=\text{p}\times\exp\left(\sum_{n=1}^{N}w_{n}\cdot\log\text{precision}_{n}\right)$

Where:

\- $p$ is the penalty term,

\- $N$ is the maximum n-gram order considered.

\- $w_{i}$ is the weights for different n-gram orders, and

\- $precision_{n}$ is the precision for n-grams.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation): ROUGE considers the
overlap of n-grams, word sequences, and subsequences between generated and
reference texts [93]. Like BLEU, ROUGE focuses on text overlap, which may not
consider deeper semantics. It can be computed using the formula below.

$\text{ROUGE}=\frac{\text{Number of overlapping units in reference and
candidate}}{\text{Number of units in reference}}$

“units” can be words, n-grams, or other text elements.

Adversarial Success Rate (ASUCR): This metric measures the percentage of
adversarial texts that successfully manipulated the model [76]. It indicates
the percentage of adversarial examples that successfully yield the model to
generate incorrect or undesirable outputs. The Attack Success Rate in the
context of QA systems can be computed using the following formula:

$\text{ASR}=\frac{\text{Number of Adversarial Examples Producing Incorrect
Answers}}{\text{Total Number of Adversarial Examples}}\text{x}100\%$

Attack Effectiveness to Noise Ratio (ENR) The Effectiveness to Noise Ratio
(ENR) is computed by dividing the ASUCR by the average per-pixel noise
introduced to alter the image [76]. Mathematically, this is represented as:

$\text{ENR}=\frac{\text{ASUCR}}{N}$

Where $N$ corresponds to the average pixel-wise squared difference between the
noise-distorted image ($I_{0}$) and the original image ($I$), calculated
across all pixels and channels. It is designed to provide more recognition to
adversarial samples with lower noise levels than those with higher ones.

Perturbation Rate: The perturbation rate measures the proportion of tokens
(words, characters) in the adversarial example compared to the original
example. Mathematically, it can be calculated using the following formula:

$\text{Perturbation Rate}=\frac{\text{Number of Modified Tokens}}{\text{Total
Number of Tokens in Original Text}}$

A higher perturbation rate indicates a more aggressive modification of the
original input, potentially conducting more successful adversarial attacks.
However, balancing perturbation and maintaining the question’s semantic
meaning is essential.

Attack Distance ($L_{p}$): Attack distance, often denoted as $L_{p}$, measures
the magnitude of change to transform an original input into an adversarial
example. This idea is often used to quantify how ”far” the adversarial example
is from the original input. The $L_{p}$ distance between an original input
vector $x$ and an adversarial input vector $x^{\prime}$ is calculated as
follows:

$\text{$L_{p}$ distance}=||x-x^{\prime}||p$

Here, $||\cdot||p$ represents the $L_{p}$ norm, and the value of p determines
the specific norm used for measuring the distance. Lower $L_{p}$ distances
indicate that the attack is more subtle and potentially more challenging to be
detected by humans.

Perceptual Adversarial Similarity Score (PASS): The Perceptual Adversarial
Similarity Score (PASS) is a metric to assess the visual similarity between an
original image and its adversarially perturbed version [94]. It quantifies how
much an image has been altered by an adversarial attack in a way that remains
perceptually similar to the human visual system.

The Structural Similarity Index (SSIM): The Structural Similarity Index (SSIM)
is used to quantify the perceptual difference between an original, clean image
and an image that an adversarial attack has perturbed [95]. The SSIM metric
helps evaluate how much the attack has deformed the image in terms of
perceptual quality. It measures the structural information, luminance, and
contrast of the images. The formula for calculating SSIM between two images
$X$ and $Y$ is given by:

$\text{SSIM}(X,Y)=\frac{{2\mu_{X}\mu_{Y}+c_{1}}}{{\mu_{X}^{2}+\mu_{Y}^{2}+c_{1}}}\cdot\frac{{2\sigma_{XY}+c_{2}}}{{\sigma_{X}^{2}+\sigma_{Y}^{2}+c_{2}}}\cdot\frac{{\sigma_{X}\sigma_{Y}+c_{3}}}{{\sigma_{X}\sigma_{Y}+c_{3}}}$

Where:

\- $\mu_{X}$ and $\mu_{Y}$ are the average pixel intensities of images $X$ and
$Y$.

\- $\sigma_{X}$ and $\sigma_{Y}$ are the standard deviations of pixel
intensities in images $X$ and $Y$

\- $\sigma_{XY}$ is the covariance between pixel intensities of images $X$ and
$Y$.

\- $c_{1}$, $c_{2}$, and $c_{3}$ are constants that stabilize the division in
case of very small denominator values.

The Peak Signal-to-Noise Ratio (PSNR) The Peak Signal-to-Noise Ratio (PSNR) is
utilized to measure the quality of the adversarial example in terms of visual
similarity to the original image. PSNR helps quantify the extent of distortion
or noise introduced by the attack.

$\text{PSNR}(X,Y)=10\cdot\log_{10}\left(\frac{\text{MSE}(X,Y)}{L^{2}}\right)$

Where:

\- $L$ is the maximum possible pixel value (e.g., 255 for 8-bit images).

\- $\text{MSE}(X,Y)$ is the Mean Squared Error between images $X$ and $Y$.

Human evaluation: Human evaluation provides a comprehensive assessment of the
quality of the text. It includes rating the quality of adversarial questions
based on various dimensions, including fluency, relevance, contextuality, etc.

Table 8 compares various evaluation metrics commonly used in the context of
adversarial attacks. The metrics are evaluated based on their supported
modalities, advantages, disadvantages, sensitivity to attacks,
interpretability, and robustness to noise. Sensitivity to attacks examines
each metric’s sensitivity to adversarial attacks, while interpretability
reflects how easily the metric’s results can be understood and analyzed.
Moreover, robustness to noise assesses how well the metric performs when
dealing with noise introduced in the data. Please note that the examinations
are generalized and might differ depending on specific use cases. According to
the table, Adversarial Success Rate is highly sensitive to attacks,
interpretability, and highly robust to noise. Moreover, ROUGE is limited to
measuring specific linguistic aspects.

Table 8: Comparison of Evaluation Metrics for Adversarial Attacks Metric |  | Supported
---
Modalities
Advantages | Disadvantages |  | Sensitivity
---
to Attacks
Interpretability |  | Robustness
---
to Noise
Perplexity | Text-based |  | Measures
---
model uncertainty
| Sensitive
---
to text length
High | High | High
Accuracy | Various |  | Simple
---
interpretation
| May not capture
---
fine-grained differences
Medium | High | High
ROUGE | Text-based |  | Focuses
---
on recall
| Limited to measuring
---
specific linguistic aspects
Medium | Medium | Medium
BLEU | Text-based |  | Measures
---
n-gram overlap
| Ignores
---
word order and context
Medium | Medium | Medium
| Adversarial
---
Success Rate
Various |  | Indicates
---
attack effectiveness
| May not account
---
for perceptual changes
High | High | High
| Perturbation
---
Rate
Image-based |  | Quantifies
---
perturbation amount
| Ignores
---
perceptual differences
Medium | Medium | High
| Attack
---
Distance
Image-based |  | Measures
---
image distortion
| Lacks
---
human interpretability
High | Medium | Medium
| Perceptual
---
Adversarial Similarity
Score (PASS)
Image-based |  | Incorporates
---
human perception
| May require
---
subjective assessment
High | Medium | Medium
| Structural
---
Similarity Index
(SSIM)
Image-based |  | Perceptually
---
relevant
| Sensitive to
---
compression artifacts
Medium | Medium | Medium
| Peak
---
Signal-to-Noise
Ratio (PSNR)
Image-based |  | Simple
---
interpretation
| Ignores
---
perceptual differences
Medium | Medium | Medium

##  7 Discussion of potential future research directions

This section highlights some of the most promising directions researchers
could tackle to improve QA systems’ robustness, reliability, and privacy
concerning adversarial challenges.

Robustness and Defense Mechanisms: Future research may focus on building more
robust QA systems resistant to adversarial attacks. It includes exploring the
defense mechanisms that detect the impact of adversarial examples. One
strategy could be to train question generation models with adversarial
examples during the training phase, allowing the model to learn from these
challenging inputs and improve its robustness. Further analysis may explore
the integration of evaluation methods to measure the model’s confidence,
enabling the model to give correct answers when uncertainty is high,
potentially avoiding false or deceptive answers.

Adversarial training: Adversarial training offers a promising approach to
maintaining the robustness of QA systems against malicious attacks. Future
research may examine adversarial training techniques that go beyond data
augmentation. By training the models on a combination of clean and carefully
crafted adversarial examples, the system navigates a variety of challenging
inputs. This process allows the model to learn from adversarial examples. As a
result, the model becomes more capable of handling adversarial attacks and
becomes resistant to manipulation and corruption of input data. Exploring
different training approaches, such as curriculum learning, can facilitate
gradual exposure to increasingly complex adversarial examples during training.
By systematically expanding the difficulty of adversarial examples, the model
learns to adapt to various adversarial examples, thereby improving its overall
robustness and ability to generalize.

Multimodal Adversarial Attacks: Multimodal adversarial attacks purpose to
manipulate textual and visual components of QA systems that utilize multimodal
inputs, such as text, images, and videos. These attacks could involve
modifying the content of the text or image to mislead the QA system into
generating incorrect or biased responses. Adversarial examples in the
multimodal setting may require more complex manipulations, as the attack needs
to impact multiple modalities while maintaining contextual meaning.

Human Defense Mechanisms in the Loop: Leveraging human evaluators as an
integral component of defense mechanisms can enhance the robustness of QA
systems. Future research may investigate ways in which human evaluators are
integrated into the loop to analyze and validate ambiguous or adversarial
queries. By combining human reasoning, QA systems can enhance their ability to
better identify and react to adversarial examples by leveraging human
insights.

Privacy Protection Defenses Adversarial attacks can compromise user privacy by
exploiting vulnerabilities in QA systems. Future research may examine
adversarial threats and defense mechanisms that reduce user privacy.
Investigating techniques such as differential privacy [96] or secure
computation [97] can be instrumental in developing QA systems to preserve the
privacy of sensitive information.

Acknowledgments G.Yigit is supported by TUBITAK - BIDEB 2211/A national
fellowship program for Ph.D. studies.

*Declarations The authors declare that they have no conflict of interest.

*Data Availability Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

## References

  *   * Szegedy et al. [2013] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
  * Kurakin et al. [2018] Kurakin, A., Goodfellow, I.J., Bengio, S.: Adversarial examples in the physical world. In: Artificial Intelligence Safety and Security, pp. 99–112. Chapman and Hall/CRC, ??? (2018)
  * Eykholt et al. [2018] Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., Song, D.: Robust physical-world attacks on deep learning visual classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625–1634 (2018)
  * Carlini et al. [2016] Carlini, N., Mishra, P., Vaidya, T., Zhang, Y., Sherr, M., Shields, C., Wagner, D., Zhou, W.: Hidden voice commands. In: 25th USENIX Security Symposium (USENIX Security 16), pp. 513–530 (2016)
  * Zhang et al. [2017] Zhang, G., Yan, C., Ji, X., Zhang, T., Zhang, T., Xu, W.: Dolphinattack: Inaudible voice commands. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 103–117 (2017)
  * Le et al. [2014] Le, N.-T., Kojiri, T., Pinkwart, N.: Automatic question generation for educational applications–the state of art. In: Advanced Computational Methods for Knowledge Engineering: Proceedings of the 2nd International Conference on Computer Science, Applied Mathematics and Applications (ICCSAMA 2014), pp. 325–338 (2014). Springer
  * Divate and Salgaonkar [2017] Divate, M., Salgaonkar, A.: Automatic question generation approaches and evaluation techniques. Current Science, 1683–1691 (2017)
  * Ch and Saha [2018] Ch, D.R., Saha, S.K.: Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies 13(1), 14–25 (2018)
  * Amidei et al. [2018] Amidei, J., Piwek, P., Willis, A.: Evaluation methodologies in automatic question generation 2013-2018 (2018)
  * Pan et al. [2019] Pan, L., Lei, W., Chua, T.-S., Kan, M.-Y.: Recent advances in neural question generation. arXiv preprint arXiv:1905.08949 (2019)
  * Patil and Patwardhan [2020] Patil, C., Patwardhan, M.: Visual question generation: The state of the art. ACM Computing Surveys (CSUR) 53(3), 1–22 (2020)
  * Kurdi et al. [2020] Kurdi, G., Leo, J., Parsia, B., Sattler, U., Al-Emari, S.: A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education 30, 121–204 (2020)
  * Das et al. [2021] Das, B., Majumder, M., Phadikar, S., Sekh, A.A.: Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning 16(1), 1–15 (2021)
  * Mulla and Gharpure [2023] Mulla, N., Gharpure, P.: Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications. Progress in Artificial Intelligence 12(1), 1–32 (2023)
  * Mishra and Jain [2016] Mishra, A., Jain, S.K.: A survey on question answering systems with classification. Journal of King Saud University-Computer and Information Sciences 28(3), 345–361 (2016)
  * Jin et al. [2023] Jin, S., Lian, X., Jung, H., Park, J., Suh, J.: Building a deep learning-based qa system from a cqa dataset. Decision Support Systems, 114038 (2023)
  * Abdel-Nabi et al. [2023] Abdel-Nabi, H., Awajan, A., Ali, M.Z.: Deep learning-based question answering: a survey. Knowledge and Information Systems 65(4), 1399–1485 (2023)
  * Wu et al. [2020] Wu, L., Wu, P., Zhang, X.: A seq2seq-based approach to question answering over knowledge bases. In: Semantic Technology: 9th Joint International Conference, JIST 2019, Hangzhou, China, November 25–27, 2019, Revised Selected Papers 9, pp. 170–181 (2020). Springer
  * Fan et al. [2019] Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., Auli, M.: Eli5: Long form question answering. arXiv preprint arXiv:1907.09190 (2019)
  * Yigit and Amasyali [2019] Yigit, G., Amasyali, M.F.: Ask me: A question answering system via dynamic memory networks. In: 2019 Innovations in Intelligent Systems and Applications Conference (ASYU), pp. 1–5 (2019). IEEE
  * Xiao et al. [2018] Xiao, L., Wang, N., Yang, G.: A reading comprehension style question answering model based on attention mechanism. In: 2018 IEEE 29th International Conference on Application-specific Systems, Architectures and Processors (ASAP), pp. 1–4 (2018). IEEE
  * Schwartz et al. [2017] Schwartz, I., Schwing, A., Hazan, T.: High-order attention models for visual question answering. Advances in Neural Information Processing Systems 30 (2017)
  * Shen et al. [2022] Shen, Y., Lai, E.M.-K., Mohaghegh, M.: Effects of similarity score functions in attention mechanisms on the performance of neural question answering systems. Neural Processing Letters 54(3), 2283–2302 (2022)
  * Chu-Carroll et al. [2003] Chu-Carroll, J., Czuba, K., Prager, J., Ittycheriah, A.: In question answering, two heads are better than one. In: Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 24–31 (2003)
  * Chekalina and Panchenko [2023] Chekalina, V., Panchenko, A.: Retrieving comparative arguments using ensemble methods and neural information retrieval. arXiv preprint arXiv:2305.01513 (2023)
  * Yu et al. [2018] Yu, J., Qiu, M., Jiang, J., Huang, J., Song, S., Chu, W., Chen, H.: Modelling domain relationships for transfer learning on retrieval-based question answering systems in e-commerce. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 682–690 (2018)
  * Bornea et al. [2021] Bornea, M., Pan, L., Rosenthal, S., Florian, R., Sil, A.: Multilingual transfer learning for qa using translation as data augmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 12583–12591 (2021)
  * Yigit and Amasyali [2023] Yigit, G., Amasyali, M.F.: Enhancing multiple-choice question answering through sequential fine-tuning and curriculum learning strategies. Knowledge and Information Systems, 1–18 (2023)
  * Kaiser et al. [2021] Kaiser, M., Saha Roy, R., Weikum, G.: Reinforcement learning from reformulations in conversational question answering over knowledge graphs. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 459–469 (2021)
  * Hua et al. [2020] Hua, Y., Li, Y.-F., Haffari, G., Qi, G., Wu, T.: Few-shot complex knowledge base question answering via meta reinforcement learning. arXiv preprint arXiv:2010.15877 (2020)
  * Misu et al. [2012] Misu, T., Georgila, K., Leuski, A., Traum, D.: Reinforcement learning of question-answering dialogue policies for virtual museum guides. In: Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 84–93 (2012)
  * Shao et al. [2023] Shao, Z., Yu, Z., Wang, M., Yu, J.: Prompting large language models with answer heuristics for knowledge-based visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14974–14983 (2023)
  * Jiang et al. [2021] Jiang, Z., Araki, J., Ding, H., Neubig, G.: How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9, 962–977 (2021)
  * Chappuis et al. [2022] Chappuis, C., Zermatten, V., Lobry, S., Le Saux, B., Tuia, D.: Prompt-rsvqa: Prompting visual context to a language model for remote sensing visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1372–1381 (2022)
  * Kumar et al. [2021] Kumar, V., Maheshwary, R., Pudi, V.: Adversarial examples for evaluating math word problem solvers. arXiv preprint arXiv:2109.05925 (2021)
  * Wang and Bansal [2018] Wang, Y., Bansal, M.: Robust machine comprehension models via adversarial training. arXiv preprint arXiv:1804.06473 (2018)
  * Jia and Liang [2017] Jia, R., Liang, P.: Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328 (2017)
  * Liu et al. [2020] Liu, K., Liu, X., Yang, A., Liu, J., Su, J., Li, S., She, Q.: A robust adversarial training approach to machine reading comprehension. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8392–8400 (2020)
  * Rosenthal et al. [2021] Rosenthal, S., Bornea, M., Sil, A.: Are multilingual bert models robust? a case study on adversarial attacks for multilingual question answering. arXiv preprint arXiv:2104.07646 (2021)
  * Cao et al. [2022] Cao, Y., Li, D., Fang, M., Zhou, T., Gao, J., Zhan, Y., Tao, D.: Tasa: Deceiving question answering models by twin answer sentences attack. arXiv preprint arXiv:2210.15221 (2022)
  * Xue et al. [2020] Xue, M., Yuan, C., Wang, J., Liu, W.: Dpaeg: a dependency parse-based adversarial examples generation method for intelligent q&a robots. Security and Communication Networks 2020, 1–15 (2020)
  * [42] Rahurkar, P., Olson, M., Tadepalli, P.: Human adversarial qa: Did the model understand the paragraph?
  * Chan et al. [2021] Chan, A., Ma, L., Juefei-Xu, F., Ong, Y.-S., Xie, X., Xue, M., Liu, Y.: Breaking neural reasoning architectures with metamorphic relation-based adversarial examples. IEEE Transactions on Neural Networks and Learning Systems 33(11), 6976–6982 (2021)
  * Lin et al. [2021] Lin, J., Zou, J., Ding, N.: Using adversarial attacks to reveal the statistical bias in machine reading comprehension models. arXiv preprint arXiv:2105.11136 (2021)
  * Sun et al. [2020] Sun, L., Hashimoto, K., Yin, W., Asai, A., Li, J., Yu, P., Xiong, C.: Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003.04985 (2020)
  * Feng et al. [2018] Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., Boyd-Graber, J.: Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.07781 (2018)
  * Zhu et al. [2020] Zhu, Y., Zhou, Y., Xia, M.: Generating semantically valid adversarial questions for tableqa. arXiv preprint arXiv:2005.12696 (2020)
  * Wang et al. [2019] Wang, B., Pei, H., Pan, B., Chen, Q., Wang, S., Li, B.: T3: Tree-autoencoder constrained adversarial text generation for targeted attack. arXiv preprint arXiv:1912.10375 (2019)
  * Gan and Ng [2019] Gan, W.C., Ng, H.T.: Improving the robustness of question answering systems to question paraphrasing. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6065–6075 (2019)
  * Fang and Wang [2023] Fang, X., Wang, W.: Defending machine reading comprehension against question-targeted attacks. In: 2023 International Joint Conference on Neural Networks (IJCNN), pp. 01–08 (2023). IEEE
  * Blohm et al. [2018] Blohm, M., Jagfeld, G., Sood, E., Yu, X., Vu, N.T.: Comparing attention-based convolutional and recurrent neural networks: Success and limitations in machine reading comprehension. arXiv preprint arXiv:1808.08744 (2018)
  * Kai et al. [2021] Kai, S., Wu, L., Tang, S., Zhuang, Y., Ding, Z., Xiao, Y., Long, B., et al.: Learning to generate visual questions with noisy supervision. Advances in Neural Information Processing Systems 34, 11604–11617 (2021)
  * Su et al. [2021] Su, H.-T., Chang, C.-H., Shen, P.-W., Wang, Y.-S., Chang, Y.-L., Chang, Y.-C., Cheng, P.-J., Hsu, W.H.: End-to-end video question-answer generation with generator-pretester network. IEEE Transactions on Circuits and Systems for Video Technology 31(11), 4497–4507 (2021)
  * Jain et al. [2017] Jain, U., Zhang, Z., Schwing, A.G.: Creativity: Generating diverse questions using variational autoencoders. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6485–6494 (2017)
  * Akbar et al. [2023] Akbar, M.F., Al Faraby, S., Romadhony, A., Adiwijaya, A.: Multimodal question generation using multimodal adaptation gate (mag) and bert-based model. In: 2023 IEEE 8th International Conference for Convergence in Technology (I2CT), pp. 1–7 (2023). IEEE
  * Huang et al. [2021] Huang, F., Jolfaei, A., Bashir, A.K.: Robust multimodal representation learning with evolutionary adversarial attention networks. IEEE Transactions on Evolutionary Computation 25(5), 856–868 (2021)
  * Lin et al. [2021] Lin, X., Bertasius, G., Wang, J., Chang, S.-F., Parikh, D., Torresani, L.: Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7005–7015 (2021)
  * Xie et al. [2021] Xie, J., Cai, Y., Huang, Q., Wang, T.: Multiple objects-aware visual question generation. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4546–4554 (2021)
  * Li et al. [2018] Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., Zhou, M.: Visual question generation as dual task of visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6116–6124 (2018)
  * Mostafazadeh et al. [2016] Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L.: Generating natural questions about an image. arXiv preprint arXiv:1603.06059 (2016)
  * Wang et al. [2020] Wang, Y.-S., Su, H.-T., Chang, C.-H., Liu, Z.-Y., Hsu, W.H.: Video question generation via semantic rich cross-modal self-attention networks learning. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2423–2427 (2020). IEEE
  * Liu et al. [2022] Liu, A., Xie, H., Liu, X., Yin, Z., Liu, S.: Revisiting audio visual scene-aware dialog. Neurocomputing 496, 227–237 (2022)
  * Goodfellow et al. [2014] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  * Kang et al. [2023] Kang, G.-C., Kim, S., Kim, J.-H., Kwak, D., Zhang, B.-T.: The dialog must go on: Improving visual dialog via generative self-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6746–6756 (2023)
  * Patel et al. [2021] Patel, A., Bindal, A., Kotek, H., Klein, C., Williams, J.: Generating natural questions from images for multimodal assistants. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2270–2274 (2021). IEEE
  * Vedd et al. [2021] Vedd, N., Wang, Z., Rei, M., Miao, Y., Specia, L.: Guiding visual question generation. arXiv preprint arXiv:2110.08226 (2021)
  * Krishna et al. [2019] Krishna, R., Bernstein, M., Fei-Fei, L.: Information maximizing visual question generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2008–2018 (2019)
  * Guo et al. [2020] Guo, Z., Zhao, Z., Jin, W., Wei, Z., Yang, M., Wang, N., Yuan, N.J.: Multi-turn video question generation via reinforced multi-choice attention network. IEEE Transactions on Circuits and Systems for Video Technology 31(5), 1697–1710 (2020)
  * Tang et al. [2020] Tang, R., Ma, C., Zhang, W.E., Wu, Q., Yang, X.: Semantic equivalent adversarial data augmentation for visual question answering. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pp. 437–453 (2020). Springer
  * Sun et al. [2023] Sun, Y., Ochiai, H., Sakuma, J.: Instance-level trojan attacks on visual question answering via adversarial learning in neuron activation space. arXiv preprint arXiv:2304.00436 (2023)
  * Liu et al. [2023] Liu, J., Lau, C.P., Chellappa, R.: Diffprotect: Generate adversarial examples with diffusion models for facial privacy protection. arXiv preprint arXiv:2305.13625 (2023)
  * Chaturvedi and Garain [2020a] Chaturvedi, A., Garain, U.: Mimic and fool: A task-agnostic adversarial attack. IEEE transactions on neural networks and learning systems 32(4), 1801–1808 (2020)
  * Chaturvedi and Garain [2020b] Chaturvedi, A., Garain, U.: Attacking vqa systems via adversarial background noise. IEEE Transactions on Emerging Topics in Computational Intelligence 4(4), 490–499 (2020)
  * Gao et al. [2021] Gao, L., Huang, Z., Song, J., Yang, Y., Shen, H.T.: Push & pull: Transferable adversarial examples with attentive attack. IEEE Transactions on Multimedia 24, 2329–2338 (2021)
  * Xu et al. [2018] Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., Song, D.: Fooling vision and language models despite localization and attention mechanism. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4951–4961 (2018)
  * Sharma et al. [2018] Sharma, V., Kalra, A., Vaibhav, S.C., Patel, L., Morency, L.-P.: Attend and attack: Attention guided adversarial attacks on visual question answering models. In: Proc. Conf. Neural Inf. Process. Syst. Workshop Secur. Mach. Learn, vol. 2 (2018)
  * Rao and Daumé III [2019] Rao, S., Daumé III, H.: Answer-based adversarial training for generating clarification questions. arXiv preprint arXiv:1904.02281 (2019)
  * Liu et al. [2020] Liu, Y., Zhang, X., Huang, F., Cheng, L., Li, Z.: Adversarial learning with multi-modal attention for visual question answering. IEEE Transactions on Neural Networks and Learning Systems 32(9), 3894–3908 (2020)
  * Ramakrishnan et al. [2018] Ramakrishnan, S., Agrawal, A., Lee, S.: Overcoming language priors in visual question answering with adversarial regularization. Advances in Neural Information Processing Systems 31 (2018)
  * Grand and Belinkov [2019] Grand, G., Belinkov, Y.: Adversarial regularization for visual question answering: Strengths, shortcomings, and side effects. arXiv preprint arXiv:1906.08430 (2019)
  * Gan et al. [2020] Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems 33, 6616–6628 (2020)
  * Feng et al. [2023] Feng, J., Sun, J., Shao, D., Cui, J.: Improving the robustness of machine reading comprehension via contrastive learning. Applied Intelligence 53(8), 9103–9114 (2023)
  * Wallace et al. [2019] Wallace, E., Rodriguez, P., Feng, S., Yamada, I., Boyd-Graber, J.: Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. Transactions of the Association for Computational Linguistics 7, 387–401 (2019)
  * Sheng et al. [2021] Sheng, S., Singh, A., Goswami, V., Magana, J., Thrush, T., Galuba, W., Parikh, D., Kiela, D.: Human-adversarial visual question answering. Advances in Neural Information Processing Systems 34, 20346–20359 (2021)
  * Zellers et al. [2019] Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., Choi, Y.: Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 (2019)
  * Dua et al. [2019] Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., Gardner, M.: Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161 (2019)
  * Zellers et al. [2018] Zellers, R., Bisk, Y., Schwartz, R., Choi, Y.: Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326 (2018)
  * Min et al. [2020] Min, S., Michael, J., Hajishirzi, H., Zettlemoyer, L.: Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645 (2020)
  * Cobbe et al. [2021] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)
  * Patel et al. [2021] Patel, A., Bhattamishra, S., Goyal, N.: Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 (2021)
  * Jelinek et al. [1977] Jelinek, F., Mercer, R.L., Bahl, L.R., Baker, J.K.: Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America 62(S1), 63–63 (1977)
  * Papineni et al. [2002] Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
  * Lin [2004] Lin, C.-Y.: Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
  * Rozsa et al. [2016] Rozsa, A., Rudd, E.M., Boult, T.E.: Adversarial diversity and hard positive generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 25–32 (2016)
  * Brunet et al. [2011] Brunet, D., Vrscay, E.R., Wang, Z.: On the mathematical properties of the structural similarity index. IEEE Transactions on Image Processing 21(4), 1488–1499 (2011)
  * Dwork [2006] Dwork, C.: Differential privacy. In: International Colloquium on Automata, Languages, and Programming, pp. 1–12 (2006). Springer
  * Lindell [2005] Lindell, Y.: Secure multiparty computation for privacy preserving data mining. In: Encyclopedia of Data Warehousing and Mining, pp. 1005–1009. IGI global, ??? (2005)