# AI-Protected Blockchain-based IoT environments: Harnessing the Future of
Network Security and Privacy

Ali Mohammadi Ruzbahani
University of Calgary
Email<EMAIL_ADDRESS>
###### Abstract

Integrating blockchain technology with the Internet of Things offers
transformative possibilities for enhancing network security and privacy in the
contemporary digital landscape, where interconnected devices and expansive
networks are ubiquitous. This paper explores the pivotal role of artificial
intelligence in bolstering blockchain-enabled IoT systems, potentially marking
a significant leap forward in safeguarding data integrity and confidentiality
across networks. Blockchain technology provides a decentralized and immutable
ledger, ideal for the secure management of device identities and transactions
in IoT networks. When coupled with AI, these systems gain the ability to not
only automate and optimize security protocols but also adaptively respond to
new and evolving cyber threats. This dual capability enhances the resilience
of networks against cyber-attacks, a critical consideration as IoT devices
increasingly permeate critical infrastructures. The synergy between AI and
blockchain in IoT is profound. AI algorithms can analyze vast amounts of data
from IoT devices to detect patterns and anomalies that may signify security
breaches. Concurrently, blockchain can ensure that data records are tamper-
proof, enhancing the reliability of AI-driven security measures. Moreover,
this research evaluates the implications of AI-enhanced blockchain systems on
privacy protection within IoT networks. IoT devices often collect sensitive
personal data, making privacy a paramount concern. AI can facilitate the
development of new protocols that ensure data privacy and user anonymity
without compromising the functionality of IoT systems. Through comprehensive
analysis and case studies, this paper aims to provide an in-depth
understanding of how AI-enhanced blockchain technology can revolutionize
network security and privacy in IoT environments.

###### Index Terms:

Blockchain, Security, IoT, AI, Privacy.

##  I Introduction

The modern digital landscape is marked by the integration of innovative
technologies that have fundamentally altered various aspects of human life and
industry [1, 2]. The advent of blockchain technology has been particularly
transformative, providing a decentralized platform that enhances security and
trust without the need for a central authority[3, 4, 2]. Its core
characteristics—decentralization, transparency, security, and
immutability—have been widely recognized for their potential to revolutionize
multiple sectors [5, 6]. Blockchain technology’s evolution from powering
cryptocurrencies to enabling complex decentralized applications (DApps)
through distributed ledgers has been significant [7, 8, 9, 10]. This
technology is now poised to play a crucial role in the Internet of Things
(IoT) by ensuring more secure and private connections between devices. The
combination of blockchain with artificial intelligence (AI) in IoT
environments presents an exciting frontier [11, 12, 13]. AI’s capability to
enhance blockchain operations through intelligent, automated, and adaptive
security responses can significantly increase the resilience of IoT networks
against cyber threats [14, 15, 16].

The focus of this paper is to explore how AI can be integrated with blockchain
technology to develop robust security solutions that ensure both the security
and privacy of data in IoT frameworks [17, 18]. This involves a detailed
examination of AI-driven security protocols, innovative data privacy measures,
and the overall implications of these technologies on the digital ecosystem.
The integration of AI with blockchain in IoT not only promises enhanced
security features but also opens up new avenues for privacy preservation in
interconnected devices [19, 20]. Through this study, we aim to provide
insights and strategic guidance on implementing AI-protected blockchain-based
systems in IoT, contributing to the advancement of network security and
privacy in an increasingly digital world [21, 22, 23]. AI applications in this
context serve multiple functions. They can automate complex processes, analyze
large streams of IoT data to detect anomalies and potential threats, and
optimize the performance and efficiency of blockchain networks. Machine
learning algorithms, a subset of AI, are particularly adept at pattern
recognition, which can be applied to predict and counteract emerging security
threats in real-time [24]. Additionally, AI can manage and enforce
blockchain’s consensus protocols more dynamically, ensuring a more responsive
and adaptive network [25, 26].

Privacy concerns in IoT are paramount as these devices often collect and
transmit personal data across networks. Integrating blockchain and AI can
enhance privacy through mechanisms such as smart contracts and advanced
encryption techniques. These tools can automate the execution of agreements
and protect data transactions, ensuring they are secure from unauthorized
access and tampering [27]. Furthermore, AI can enhance blockchain’s capability
to offer decentralized privacy-preserving technologies such as zero-knowledge
proofs, allowing IoT devices to validate transactions without revealing any
underlying personal data [28, 29]. The potential for these integrated
technologies to transform IoT security and privacy extends into various
industries, including healthcare, automotive, and smart homes, where the
integrity and confidentiality of data are crucial. In healthcare, for example,
ensuring the privacy and security of patient data while utilizing IoT devices
for monitoring and treatment can significantly enhance patient trust and
treatment outcomes [30, 31]. Similarly, in the automotive sector, blockchain
and AI can secure vehicular communication systems, making connected vehicles
safer from cyber-attacks and privacy breaches [32].

In conclusion, the integration of AI and blockchain technology holds the
potential to significantly fortify IoT environments against security threats
and privacy issues. This paper has explored the synergistic effects of these
technologies and proposed a framework for their implementation in IoT systems.
Moving forward, it is essential for industry stakeholders and regulators to
consider these findings in their strategic planning and policy-making to
ensure the secure and ethical use of IoT technologies in a digitally connected
world [33, 34, 35, 36].

TABLE I: Comparison of Existing Surveys on AI-Enhanced Blockchain in IoT Reference | Investigated Challenges | Role of Blockchain | Role of AI | Security Enhancement |  Remark
---|---|---|---|---|---
[37] | ✓ | ✓ | ✗ | ✗ |  Focused on security challenges in blockchain applications.
[38] | ✓ | ✓ | ✗ | ✗ |  Reviewed AI applications for threat detection in IoT.
[39] | ✗ | ✓ | ✓ | ✗ |  Discussed AI-driven blockchain architectures for IoT.
[40] | ✗ | ✓ | ✓ | ✓ |  Explored AI-enhanced security solutions in blockchain IoT frameworks.
[41] | ✓ | ✗ | ✓ | ✗ |  Analyzed challenges in integrating AI with blockchain for IoT.

###  I-A Existing Surveys

As the integration of blockchain technology and artificial intelligence (AI)
continues to redefine the landscape of Internet of Things (IoT) environments,
scholarly interest has increasingly focused on how these technologies can
collectively enhance network security and privacy [15, 42, 24]. Existing
research has typically examined the capabilities and challenges of blockchain
and AI within IoT systems in isolation. For example, numerous studies have
investigated how the inherent security features of blockchain can safeguard
IoT devices and data, while others have scrutinized AI’s role in real-time
detection and response to security threats [25, 43, 28, 44]. A pivotal
systematic literature review highlighted the current applications and security
solutions provided by blockchain in IoT, emphasizing the need for advanced
mechanisms to counter dynamic and sophisticated cyber threats [37].
Concurrently, another significant study assessed advancements in AI algorithms
capable of autonomously detecting anomalies and potential breaches in IoT
networks, proposing frameworks for integrating these algorithms with
blockchain technologies [38]. Further research has proposed architectural
frameworks that meld AI and blockchain to boost the security and efficiency of
IoT systems [39]. These frameworks strive to capitalize on AI’s predictive
capabilities and blockchain’s tamper-resistant properties to forge a secure
and resilient digital infrastructure.

Moreover, surveys such as [40] have delved into multi-layered security
approaches that amalgamate the strengths of AI and blockchain. These
approaches present comprehensive protection strategies for interconnected
devices across various sectors, outlining the layers of security that can be
enhanced through the integration of AI and blockchain. However, in contrast to
the existing literature that often treats AI and blockchain as distinct and
separate entities, our study takes a novel approach by exploring their
synergistic integration within IoT environments. This paper aims to offer a
holistic view of how combined AI and blockchain technologies can revolutionize
network security and privacy, marking a significant advancement in the field
[32, 30, 45]. By leveraging AI’s machine learning capabilities to analyze data
patterns and predict potential threats, and combining this with blockchain’s
ability to secure data through its immutable ledger system, we can create a
robust framework for IoT security. This integration not only enhances the
detection of anomalies but also ensures the integrity and confidentiality of
the data being transmitted across IoT devices.

Furthermore, the paper discusses the implications of this integrated approach
on privacy concerns, a critical aspect as IoT devices pervasively collect and
process personal data. The use of blockchain can provide a decentralized
approach to data management, reducing the risks associated with central data
storage, while AI can enhance data processing and control access, ensuring
that data privacy is maintained without sacrificing efficiency. In addition to
technical enhancements, our study also addresses the policy and regulatory
challenges that arise with the deployment of AI and blockchain in IoT. It
advocates for the development of clear guidelines and standards to govern the
ethical use of these technologies, ensuring that they contribute positively to
societal needs while safeguarding against misuse.

In conclusion, the synergy between AI and blockchain holds the potential to
transform IoT security and privacy fundamentally. Our research not only maps
out the technological advancements but also provides strategic guidance for
future implementations, aiming to foster an environment where technology
serves as a cornerstone for secure, efficient, and trustworthy digital
interactions.

###  I-B Contributions

This survey makes significant contributions to the domain of network and
service security by focusing on the integration of AI with blockchain
technology within IoT environments. We address a gap in the current literature
by providing an exhaustive analysis of how AI can enhance blockchain
applications in IoT, ensuring more secure and private networks. Our study
distinguishes itself by examining a broad spectrum of security mechanisms and
vulnerabilities specific to IoT environments, exploring how AI-enhanced
blockchain solutions can address these challenges.

By evaluating real-world applications and potential future developments, this
survey serves as a vital resource for both researchers and practitioners. It
offers a comprehensive understanding of the current state of IoT security, the
role of AI in enhancing these frameworks, and the transformative potential of
blockchain technology in creating more robust digital networks.

###  I-C Paper Organization

The structure of this survey begins with Section II, which outlines the
methodological approach and presents the key research questions. Section III
investigates the current landscape of IoT security challenges and how
blockchain technology can address them. Section IV delves into AI’s role in
enhancing security protocols within blockchain-based IoT systems. Section V
explores the practical applications and security frameworks that combine AI
and blockchain in IoT. Section VIII discusses the implications of these
technologies on future network security and privacy. The final section, VII,
synthesizes the findings and discusses potential directions for future
research in AI-protected blockchain-based IoT environments. Concluding remarks
are provided in Section IX.

TABLE II: Cyber Attacks on AI-Enhanced Blockchain in IoT Attack Type |  Method |  Impact |  Objective
---|---|---|---
Network Intrusions |  Unauthorized entry |  Undermines network integrity |  Seeks unauthorized access and control
Data Breaches |  Leveraging security flaws |  Reveals confidential data |  Aims to steal or alter private information
Man-in-the-Middle Attacks |  Interfering with data flows |  Modifies or intercepts data during transmission |  Targets the acquisition and alteration of sensitive data
Ransomware Attacks |  Holding data hostage for ransom |  Disrupts operations |  Demands payment in exchange for access
AI Poisoning |  Corrupting machine learning data |  Misdirects AI decision-making |  Seeks to alter AI operations to benefit the attacker
Side Channel Attacks |  Utilizing indirect system information |  Accesses critical operational data |  Obtains sensitive data through indirect means
Sybil Attacks |  Creating numerous fictitious identities |  Compromises consensus mechanisms |  Affects decision-making or alters transaction records
Distributed Denial of Service (DDoS) Attacks |  Bombarding the network with excessive requests |  Overwhelms system availability |  Halts IoT functions and services

##  II Research Methodology and Questions

Our research methodology rigorously investigates the integration of artificial
intelligence (AI) with blockchain technology in IoT settings, focusing on
boosting network security and privacy. We adopt a qualitative research
strategy, compiling and analyzing data from a broad range of academic papers
and formulating specific metrics to address several critical research
questions. This approach enables us to assess the current landscape of
blockchain and AI utilization in IoT, explore the integration challenges they
face, and identify potential strategies for enhancing these technologies to
create stronger security frameworks [27, 46, 32]. The research is guided by
the following key questions:

  * •

RQ1: What are the existing uses of blockchain and AI in strengthening IoT
security?

  * •

RQ2: What are the main challenges and solutions associated with merging AI
with blockchain technology in IoT contexts?

  * •

RQ3: In what ways can AI amplify the effectiveness of blockchain-driven
security mechanisms in IoT?

To address these questions, we have identified pertinent research papers,
including journal articles, conference proceedings, and other publications
that describe different techniques, frameworks, and methodologies used in
enhancing IoT security through blockchain and AI integration. The subsequent
sections discuss the findings for each research question.

##  III Cyber Attacks on IoT Systems

IoT systems, while transformative, are susceptible to a range of cyber threats
that underscore the need for advanced security measures. These threats include
network intrusions, data breaches, and other sophisticated cyber attacks that
exploit vulnerabilities in IoT devices and networks[47]. AI-enhanced
blockchain solutions offer promising avenues for defending against these
threats by providing decentralized security mechanisms and intelligent threat
detection and response systems. This section outlines various types of cyber
attacks common in IoT environments and discusses how blockchain and AI
technologies can mitigate these risks effectively[33, 35].

##  IV Blockchain and AI Integration in IoT Security

This section explores the dynamic field of blockchain and AI integration
within IoT security frameworks. We highlight studies that apply both
technologies to create innovative solutions for securing IoT devices and data.
This includes AI-driven anomaly detection systems that utilize blockchain for
data integrity and decentralized operations, enhancing the trustworthiness and
reliability of IoT networks. The exploration of blockchain and AI in this
context illuminates the vast potential of these technologies to provide
comprehensive security solutions that are both scalable and robust.

##  V Enhancing IoT Security with AI-Protected Blockchain Solutions

The integration of AI and blockchain holds significant promise for enhancing
IoT security. This section discusses various applications of these
technologies in IoT, such as secure communications, data privacy, and device
authentication. By leveraging AI’s ability to quickly analyze patterns and
detect anomalies, combined with blockchain’s capabilities for ensuring data
integrity and preventing tampering, these integrated solutions address key
security challenges in IoT. We review practical implementations and
theoretical models that illustrate how AI-protected blockchain solutions can
effectively secure IoT environments, fostering greater adoption and trust in
these technologies[48].

Figure 1: Overview of security challenges of SC

##  VI Role of AI-Enhanced Blockchain in IoT Network and Service Security

This subsection explores various instances where blockchain, enhanced by
artificial intelligence (AI), has been leveraged to bolster network and
service security, particularly within IoT environments. An example is the AI-
Blockchain Hybrid System, which integrates AI’s predictive and adaptive
capabilities with blockchain’s security features to enhance network resilience
and data integrity[33, 35]. This approach is designed to efficiently handle
dynamic security threats and optimize IoT device performance through real-time
data analysis and decentralized management.

In the context of improving security protocols, the study [49] introduces an
AI-driven monitoring system that employs blockchain for secure data logging
and anomaly detection in network transactions. This system provides a robust
framework for preventing and responding to cyber threats in a decentralized
manner, reducing the risk of single points of failure[46, 50].

Another crucial aspect is the use of AI to enhance blockchain’s operational
efficiency and security in diverse sectors. For instance, in healthcare, [51,
52] discusses a blockchain-based system, powered by AI, to secure health-
related transactions and manage patient data across decentralized networks.
This ensures enhanced privacy, security, and interoperability within
healthcare systems.

Furthermore, in cloud environments, [53, 54] presents a novel AI-Blockchain
architecture for cloud manufacturing that uses intelligent algorithms to
optimize Service Level Agreements (SLAs) and ensure trust among different
cloud service stakeholders. This model not only enhances security but also
improves the scalability and efficiency of cloud services.

These examples showcase the diverse applications of AI-enhanced blockchain
technology in strengthening network and service security across various
domains, particularly in IoT settings[48, 55, 56].

##  VII Future Roadmap: The Convergence of AI and Blockchain in Enhancing
Digital Security

As depicted in Fig. 2, the integration of AI with blockchain has progressively
evolved, enhancing the capabilities and efficiency of blockchain applications
in security-sensitive environments such as IoT. This roadmap illustrates the
transformative impact of AI in enabling blockchain systems to adapt and
respond to emerging security challenges dynamically.

The potential for AI to optimize blockchain technology for security purposes
is vast. Researchers are developing sophisticated AI tools to improve anomaly
detection and security management within blockchain networks. For instance,
the machine-learning-based tool, MLBP-Blockchain, discussed in [57], predicts
and identifies potential security flaws in blockchain applications, enabling
proactive security measures.

Further exploration into AI-driven security solutions for blockchain includes
techniques such as those presented in [58] and [59, 60], which utilize deep
learning and graph neural networks, respectively, to enhance the detection of
vulnerabilities in blockchain networks. These advancements provide more
precise and efficient security assessments.

Looking forward, the intersection of AI and blockchain is expected to yield
innovative solutions that not only enhance the security and efficiency of
digital networks but also support new applications, such as intelligent
automated systems in smart cities, secure supply chain management, and
privacy-preserving data sharing[61, 46]. Such developments promise to
revolutionize network and service security, paving the way for a more secure
and interconnected digital future[55, 62].

Figure 2: The evolution of adopting AI in SC

##  VIII The Influence of AI on Blockchain Technology for IoT Security

This section delves into the ongoing research into the influence of AI on
blockchain technology, specifically within IoT environments, as illustrated in
Fig. 3. Key trends include AI-enhanced security protocols, autonomous
blockchain management, AI-driven dispute resolution, and the incorporation of
bio-inspired AI algorithms. These topics are further explored in subsections
VIII-A, VIII-B, VIII-C, and VIII-D, respectively.

Figure 3: Future trends of AI in Blockchain for IoT Security

###  VIII-A AI-Enhanced Security Protocols

Artificial Intelligence (AI) is revolutionizing security management within
Internet of Things (IoT) networks by integrating enhanced capabilities through
machine learning and natural language processing. These AI tools allow for
more efficient and accurate security monitoring, significantly reducing human
error and accelerating response times. In blockchain-based IoT security
systems, the potential of AI is particularly vast. AI technologies can
automate and optimize threat detection and response processes by continuously
learning from network activities and emerging threats. This shift toward AI-
enabled solutions initiates a new era of intelligent security management where
systems can predict and mitigate risks autonomously. Furthermore, AI can
tailor security measures based on the behavior patterns and vulnerability
assessments of the network, ensuring a proactive stance against cyber threats.
This not only enhances the security of IoT devices but also ensures the
integrity and confidentiality of the data flowing through these networks. By
leveraging AI, blockchain technology can be transformed into a more dynamic
and secure framework that supports the growing complexity and scale of IoT
ecosystems.

###  VIII-B Autonomous Blockchain Management

Drawing parallels from the finance sector, where AI-driven robo-advisors have
revolutionized investment management, similar AI algorithms could be adapted
to autonomously manage blockchain operations. This adaptation would enhance
the efficiency and security of blockchain applications in IoT settings.
Autonomous AI systems would actively monitor blockchain performance,
dynamically adjust protocols, and respond to threats in real-time without
human intervention. This level of automation democratizes advanced security
strategies, previously only accessible to large-scale operators, and extends
them to smaller entities or individual IoT applications. The implementation of
such autonomous systems could lead to significant improvements in the
resilience and reliability of blockchain networks, ensuring seamless and
secure operations across various IoT applications. This shift not only
streamlines the management of blockchain networks but also opens up new
possibilities for the broader adoption and scalability of blockchain
technology in diverse sectors.

###  VIII-C AI-Driven Dispute Resolution

AI’s application in Online Dispute Resolution (ODR) systems exemplifies its
potential to facilitate quicker and fairer resolutions in blockchain
transactions, especially relevant in IoT environments. By analyzing
transaction data and contractual conditions, AI can provide objective analyses
and predict likely outcomes, thereby expediting the dispute resolution process
within decentralized networks. This application of AI in dispute resolution
not only enhances the efficiency of resolving conflicts but also increases the
transparency and trust in blockchain transactions. Automated, AI-driven
solutions can handle a high volume of disputes simultaneously, reducing the
need for human arbitrators and decreasing the overall time and cost associated
with dispute resolutions. Such capabilities are crucial in IoT scenarios where
transactions occur frequently and disputes need to be resolved swiftly to
maintain the integrity of the network.

###  VIII-D Bio-Inspired Blockchain Solutions

Bio-inspired AI technologies, such as neural networks and genetic algorithms,
present exciting opportunities for enhancing blockchain applications,
particularly in managing complex, dynamic tasks in IoT security. For instance,
swarm intelligence could be utilized to optimize resource allocation across
blockchain networks, improving efficiency and scalability. Similarly, genetic
algorithms can be applied to evolve blockchain protocols over time, enhancing
their adaptability and robustness. The incorporation of bio-inspired AI models
into blockchain frameworks allows for the simulation of natural processes that
can significantly improve decision-making and problem-solving capabilities
within these systems. Such advancements not only bolster the security features
of blockchain but also enhance its capacity to adapt to changing network
conditions and emerging threats in IoT environments.

##  IX Conclusion

In conclusion, our research underscores the transformative role of AI in
enhancing blockchain technology for the security of the IoT. AI not only
strengthens existing security protocols but also introduces innovative
capabilities such as autonomous management and adaptive dispute resolution
mechanisms. As the integration of AI with blockchain deepens, the potential
for these technologies to comprehensively safeguard and optimize IoT
ecosystems becomes increasingly significant. This paper has thoroughly
explored how AI can dynamically adapt to and effectively address the
continuously evolving security challenges in decentralized networks. By
leveraging AI’s advanced analytical and predictive capabilities, blockchain
systems can become not only more secure but also smarter, thereby promising a
future where digital transactions are both secure and intelligent. The
integration of AI with blockchain offers a dual advantage: AI’s ability to
swiftly analyze and respond to threats complements blockchain’s inherent
strengths in providing a tamper-proof and transparent transaction ledger.
Moreover, the findings of this study offer a solid foundation for future
research and development in this area. They suggest a clear path toward the
creation of an integrated, adaptive, and robust digital infrastructure for IoT
security that can effectively counteract and mitigate potential cyber threats.
This investigation highlights the importance of such advancements in the
context of increasing reliance on IoT technologies across various sectors,
including healthcare, finance, and smart cities. Furthermore, this research
serves as a roadmap for both researchers and practitioners in the field. It
encourages further exploration into the potential of AI-enhanced blockchain
technologies, aiming to leverage these advancements to secure digital networks
and systems comprehensively. By continuing to explore these technologies, we
can unlock powerful tools for enhancing data integrity, confidentiality, and
operational efficiency in an increasingly interconnected world. Ultimately,
the integration of AI with blockchain represents a significant step forward in
our approach to IoT security. It calls for ongoing innovation and
collaboration among technologists, industry leaders, and policymakers to
ensure that the deployment of these technologies maximizes benefits while
minimizing risks to privacy and security.

## References

  * [1] H. M. Rouzbahani, H. Karimipour, A. Dehghantanha, and R. M. Parizi, “Blockchain applications in power systems: A bibliometric analysis,” _Blockchain Cybersecurity, Trust and Privacy_ , pp. 129–145, 2020.
  * [2] D. Namakshenas, A. Yazdinejad, A. Dehghantanha, and G. Srivastava, “Federated quantum-based privacy-preserving threat detection model for consumer internet of things,” _IEEE Transactions on Consumer Electronics_ , 2024.
  * [3] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, Q. Zhang, and K.-K. R. Choo, “An energy-efficient sdn controller architecture for iot networks with blockchain-based security,” _IEEE Transactions on Services Computing_ , vol. 13, no. 4, pp. 625–638, 2020.
  * [4] H. M. Rouzbahani, H. Karimipour, and L. Lei, “Multi-layer defense algorithm against deep reinforcement learning-based intruders in smart grids,” _International Journal of Electrical Power & Energy Systems_, vol. 146, p. 108798, 2023.
  * [5] ——, “A review on virtual power plant for energy management,” _Sustainable energy technologies and assessments_ , vol. 47, p. 101370, 2021.
  * [6] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, and K.-K. R. Choo, “Blockchain-enabled authentication handover with efficient privacy protection in sdn-based 5g networks,” _IEEE Transactions on Network Science and Engineering_ , vol. 8, no. 2, pp. 1120–1132, 2019.
  * [7] H. M. Ruzbahani, A. Rahimnejad, and H. Karimipour, “Smart households demand response management with micro grid,” in _2019 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT)_. IEEE, 2019, pp. 1–5.
  * [8] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, M. Hammoudeh, H. Karimipour, and G. Srivastava, “Block hunter: Federated learning for cyber threat hunting in blockchain-based iiot networks,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 11, pp. 8356–8366, 2022.
  * [9] H. M. Rouzbahani, A. H. Bahrami, and H. Karimipour, “A snapshot ensemble deep neural network model for attack detection in industrial internet of things,” _AI-Enabled Threat Detection and Security Analysis for Industrial IoT_ , pp. 181–194, 2021.
  * [10] G. K. Jagarlamudi, A. Yazdinejad, R. M. Parizi, and S. Pouriyeh, “Exploring privacy measurement in federated learning,” _The Journal of Supercomputing_ , pp. 1–41, 2023.
  * [11] H. M. Rouzbahani, Z. Faraji, M. Amiri-Zarandi, and H. Karimipour, “Ai-enabled security monitoring in smart cyber physical grids,” _Security of Cyber-Physical Systems: Vulnerability and Impact_ , pp. 145–167, 2020.
  * [12] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, and H. Karimipour, “Federated learning for drone authentication,” _Ad Hoc Networks_ , vol. 120, p. 102574, 2021.
  * [13] T. Crawford, S. Duong, R. Fueston, A. Lawani, S. Owoade, A. Uzoka, R. M. Parizi, and A. Yazdinejad, “Ai in software engineering: A survey on project management applications,” _arXiv preprint arXiv:2307.15224_ , 2023.
  * [14] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, H. Karimipour, G. Srivastava, and M. Aledhari, “Enabling drones in the internet of things with decentralized blockchain-based security,” _IEEE Internet of Things Journal_ , vol. 8, no. 8, pp. 6406–6415, 2020.
  * [15] B. Zolfaghari, E. Rabieinejad, A. Yazdinejad, R. M. Parizi, and A. Dehghantanha, “Crypto makes ai evolve.”
  * [16] H. M. Rouzbahani, H. Karimipour, and L. Lei, “Optimizing scheduling policy in smart grids using probabilistic delayed double deep q-learning (p3dql) algorithm,” _Sustainable Energy Technologies and Assessments_ , vol. 53, p. 102712, 2022.
  * [17] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, G. Srivastava, and H. Karimipour, “Secure intelligent fuzzy blockchain framework: Effective threat detection in iot networks,” _Computers in Industry_ , vol. 144, p. 103801, 2023.
  * [18] H. M. Rouzbahani, H. Karimipour, E. Fraser, A. Dehghantanha, E. Duncan, A. Green, and C. Russell, “Communication layer security in smart farming: A survey on wireless technologies,” _arXiv preprint arXiv:2203.06013_ , 2022.
  * [19] A. Yazdinejad, A. Bohlooli, and K. Jamshidi, “Efficient design and hardware implementation of the openflow v1. 3 switch on the virtex-6 fpga ml605,” _The Journal of Supercomputing_ , vol. 74, pp. 1299–1320, 2018.
  * [20] M. Sisk, M. Majlis, C. Page, and A. Yazdinejad, “Analyzing xai metrics: Summary of the literature review,” _Authorea Preprints_ , 2023.
  * [21] A. Yazdinejad, R. M. Parizi, A. Bohlooli, A. Dehghantanha, and K.-K. R. Choo, “A high-performance framework for a network programmable packet processor using p4 and fpga,” _Journal of Network and Computer Applications_ , vol. 156, p. 102564, 2020.
  * [22] A. H. Bahrami and H. M. Rouzbahani, “Cyber security of smart manufacturing execution systems: a bibliometric analysis,” _AI-Enabled Threat Detection and Security Analysis for Industrial IoT_ , pp. 105–119, 2021.
  * [23] A. Yazdinejad, B. Zolfaghari, A. Dehghantanha, H. Karimipour, G. Srivastava, and R. M. Parizi, “Accurate threat hunting in industrial internet of things edge devices,” _Digital Communications and Networks_ , vol. 9, no. 5, pp. 1123–1130, 2023.
  * [24] J. Sakhnini, H. Karimipour, A. Dehghantanha, A. Yazdinejad, T. R. Gadekallu, N. Victor, and A. Islam, “A generalizable deep neural network method for detecting attacks in industrial cyber-physical systems,” _IEEE Systems Journal_ , 2023.
  * [25] A. Yazdinejad, A. Bohlooli, and K. Jamshidi, “P4 to sdnet: Automatic generation of an efficient protocol-independent packet parser on reconfigurable hardware,” in _2018 8th International Conference on Computer and Knowledge Engineering (ICCKE)_. IEEE, 2018, pp. 159–164.
  * [26] H. M. Rouzbahani, A. Dehghantanha, and K.-K. R. Choo, “Big data analytics and forensics: An overview,” _Handbook of Big Data Analytics and Forensics_ , pp. 1–5, 2022.
  * [27] B. Zolfaghari, A. Yazdinejad, A. Dehghantanha, J. Krzciok, and K. Bibak, “The dichotomy of cloud and iot: Cloud-assisted iot from a security perspective,” _arXiv preprint arXiv:2207.01590_ , 2022.
  * [28] A. Yazdinejad, A. Bohlooli, and K. Jamshidi, “Performance improvement and hardware implementation of open flow switch using fpga,” in _2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI)_. IEEE, 2019, pp. 515–520.
  * [29] H. M. Rouzbahani, H. Karimipour, and L. Lei, “Optimizing resource swap functionality in ioe-based grids using approximate reasoning reward-based adjustable deep double q-learning,” _IEEE Transactions on Consumer Electronics_ , 2023.
  * [30] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, and G. Epiphaniou, “An optimized fuzzy deep learning model for data classification based on nsga-ii,” _Neurocomputing_ , vol. 522, pp. 116–128, 2023.
  * [31] H. Mohammadi Rouzbahani, H. Karimipour, A. Rahimnejad, A. Dehghantanha, and G. Srivastava, “Anomaly detection in cyber-physical systems using machine learning,” _Handbook of big data privacy_ , pp. 219–235, 2020.
  * [32] A. Yazdinejad, A. Dehghantanha, H. Karimipour, G. Srivastava, and R. M. Parizi, “An efficient packet parser architecture for software-defined 5g networks,” _Physical Communication_ , vol. 53, p. 101677, 2022.
  * [33] A. Yazdinejad, A. Dehghantanha, and G. Srivastava, “Ap2fl: auditable privacy-preserving federated learning framework for electronics in healthcare,” _IEEE Transactions on Consumer Electronics_ , 2023.
  * [34] H. M. Rouzbahani, H. Karimipour, and L. Lei, “An ensemble deep convolutional neural network model for electricity theft detection in smart grids,” in _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, 2020, pp. 3637–3642.
  * [35] A. Yazdinejad, E. Rabieinejad, T. Hasani, and G. Srivastava, “A bert-based recommender system for secure blockchain-based cyber physical drug supply chain management,” _Cluster Computing_ , vol. 26, no. 6, pp. 3389–3403, 2023.
  * [36] H. Mohammadi Rouzbahani, H. Karimipour, and G. Srivastava, “Big data application for security of renewable energy resources,” _Handbook of Big Data Privacy_ , pp. 237–254, 2020.
  * [37] W. Zou, D. Lo, P. S. Kochhar, X.-B. D. Le, X. Xia, Y. Feng, Z. Chen, and B. Xu, “Smart contract development: Challenges and opportunities,” _IEEE Transactions on Software Engineering_ , vol. 47, no. 10, pp. 2084–2106, 2020.
  * [38] N. Sánchez-Gómez, J. Torres-Valderrama, J. A. García-García, J. J. Gutiérrez, and M. J. Escalona, “Model-based software design and testing in blockchain smart contracts: A systematic literature review,” _IEEE Access_ , vol. 8, pp. 164 556–164 569, 2020.
  * [39] S. Wang, L. Ouyang, Y. Yuan, X. Ni, X. Han, and F.-Y. Wang, “Blockchain-enabled smart contracts: Architecture, applications, and future trends,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 49, no. 11, pp. 2266–2277, 2019.
  * [40] S. Khan, M. B. Amin, A. T. Azar, and S. Aslam, “Towards interoperable blockchains: A survey on the role of smart contracts in blockchain interoperability,” _IEEE Access_ , vol. 9, pp. 116 672–116 691, 2021.
  * [41] V. Y. Kemmoe, W. Stone, J. Kim, D. Kim, and J. Son, “Recent advances in smart contracts: A technical overview and state of the art,” _IEEE Access_ , vol. 8, pp. 117 782–117 801, 2020.
  * [42] H. M. Ruzbahani and H. Karimipour, “Optimal incentive-based demand response management of smart households,” in _2018 IEEE/IAS 54th Industrial and Commercial Power Systems Technical Conference (I &CPS)_. IEEE, 2018, pp. 1–7.
  * [43] H. M. Rouzbahani, H. Karimipour, and G. Srivastava, “Big data application for renewable energy resource security,” _Handbook of Big Data and Privacy_ , 2019.
  * [44] A. Applebaum, C. Dennler, P. Dwyer, M. Moskowitz, H. Nguyen, N. Nichols, N. Park, P. Rachwalski, F. Rau, A. Webster _et al._ , “Bridging automated to autonomous cyber defense: Foundational analysis of tabular q-learning,” in _Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security_ , 2022, pp. 149–159.
  * [45] H. M. Rouzbahani _et al._ , “Energy scheduling in ioe-enabled smart grids using probabilistic delayed double deep q-learning (p3dql) algorithm,” _Authorea Preprints_ , 2023.
  * [46] H. Mohammadi Rouzbahani, “Application of internet of energy in smart grids using deep reinforcement learning,” 2022.
  * [47] D. Sheridan, J. Harris, F. Wear, J. Cowell Jr, E. Wong, and A. Yazdinejad, “Web3 challenges and opportunities for the market,” _arXiv preprint arXiv:2209.02446_ , 2022.
  * [48] K. Viswanathan and A. Yazdinejad, “Security considerations for virtual reality systems,” _arXiv preprint arXiv:2201.02563_ , 2022.
  * [49] Q. Ren, H. Liu, Y. Li, and H. Lei, “Demo: Cloak: A framework for development of confidential blockchain smart contracts,” in _Proceedings of IEEE 41st International Conference on Distributed Computing Systems (ICDCS)_ , Washington DC, USA, July 2021.
  * [50] A. Yazdinejad, A. Dehghantanha, G. Srivastava, H. Karimipour, and R. M. Parizi, “Hybrid privacy preserving federated learning against irregular users in next-generation internet of things,” _Journal of Systems Architecture_ , p. 103088, 2024.
  * [51] A. A. Khan, A. A. Wagan, A. A. Laghari, A. R. Gilal, I. A. Aziz, and B. A. Talpur, “Biomt: A state-of-the-art consortium serverless network architecture for healthcare system using blockchain smart contracts,” _IEEE Access_ , vol. 10, pp. 78 887–78 898, 2022.
  * [52] A. Yazdinejad, “Secure and private ml-based cybersecurity framework for industrial internet of things (iiot),” Ph.D. dissertation, University of Guelph, 2024.
  * [53] W. Tan, H. Zhu, J. Tan, Y. Zhao, L. D. Xu, and K. Guo, “A novel service level agreement model using blockchain and smart contract for cloud manufacturing in industry 4.0,” _Enterprise Information Systems_ , vol. 16, no. 12, p. 1939426, 2022.
  * [54] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, and K.-K. R. Choo, “P4-to-blockchain: A secure blockchain-enabled packet parser for software defined networking,” _Computers & Security_, vol. 88, p. 101629, 2020.
  * [55] S. Nakhodchi, B. Zolfaghari, A. Yazdinejad, and A. Dehghantanha, “Steeleye: An application-layer attack detection and attribution model in industrial control systems using semi-deep learning,” in _2021 18th International Conference on Privacy, Security and Trust (PST)_. IEEE, 2021, pp. 1–8.
  * [56] A. Yazdinejadna, R. M. Parizi, A. Dehghantanha, and M. S. Khan, “A kangaroo-based intrusion detection system on software-defined networks,” _Computer Networks_ , vol. 184, p. 107688, 2021.
  * [57] A. GÜl, Y. KÖorĞlu, and A. Şen, “Machine learning based bug prediction engine for smart contracts,” in _Proceedings of Turkish National Software Engineering Symposium (UYMS)_ , Istanbul, Turkey, October 2020.
  * [58] Z. Liu, P. Qian, X. Wang, Y. Zhuang, L. Qiu, and X. Wang, “Combining graph neural networks with expert knowledge for smart contract vulnerability detection,” _IEEE Transactions on Knowledge and Data Engineering (Early Access Article)_ , pp. 1–1, 2021.
  * [59] X. Yu, H. Zhao, B. Hou, Z. Ying, and B. Wu, “Deescvhunter: A deep learning-based framework for smart contract vulnerability detection,” in _Proceedings of International Joint Conference on Neural Networks (IJCNN)_ , Shenzhen, China, July 2021.
  * [60] A. Yazdinejad, M. Kazemi, R. M. Parizi, A. Dehghantanha, and H. Karimipour, “An ensemble deep learning model for cyber threat hunting in industrial internet of things,” _Digital Communications and Networks_ , vol. 9, no. 1, pp. 101–110, 2023.
  * [61] A. Yazdinejad, H. HaddadPajouh, A. Dehghantanha, R. M. Parizi, G. Srivastava, and M.-Y. Chen, “Cryptocurrency malware hunting: A deep recurrent neural network approach,” _Applied Soft Computing_ , vol. 96, p. 106630, 2020.
  * [62] M. Kazemi and A. Yazdinejad, “Towards automated benchmark support for multi-blockchain interoperability-facilitating platforms,” _arXiv preprint arXiv:2103.03866_ , 2021.


# Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with
Zero Training

Anthony Meng Huat Tiong1,2, Junnan Li1, Boyang Li2,
Silvio Savarese1, and Steven C.H. Hoi1
1Salesforce Research 2Nanyang Technological University, Singapore
{anthony.tiong, junnan.li, ssavarese<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa

###### Abstract

Visual question answering (VQA) is a hallmark of vision and language reasoning
and a challenging task under the zero-shot setting. We propose Plug-and-Play
VQA (PnP-VQA), a modular framework for zero-shot VQA. In contrast to most
existing works, which require substantial adaptation of pretrained language
models (PLMs) for the vision modality, PnP-VQA requires no additional training
of the PLMs. Instead, we propose to use natural language and network
interpretation as an intermediate representation that glues pretrained models
together. We first generate question-guided informative image captions, and
pass the captions to a PLM as context for question answering. Surpassing end-
to-end trained baselines, PnP-VQA achieves state-of-the-art results on zero-
shot VQAv2 Goyal et al. (2017) and GQA Hudson and Manning (2019). With 11B
parameters, it outperforms the 80B-parameter Flamingo model Alayrac et al.
(2022) by 8.5% on VQAv2. With 738M PLM parameters, PnP-VQA achieves an
improvement of 9.1% on GQA over FewVLM Jin et al. (2022) with 740M PLM
parameters.

##  1 Introduction

Recent years have witnessed unprecedented performance gains on many natural
language reasoning tasks, especially in zero-shot and few-shot settings, being
derived from scaling up pretrained language models (PLMs) and their training
data Devlin et al. (2019); Liu et al. (2019); Brown et al. (2020); Raffel et
al. (2020); Black et al. (2022); Sanh et al. (2022); Wei et al. (2021).
Inspired by their success, a natural thought is that utilizing PLMs should
also boost zero-shot performance in vision-language reasoning tasks.

However, to leverage PLMs for vision-language tasks, most existing methods
require non-trivial adaptation of the PLMs for the vision modality, which
necessitates the design of new network components and training objectives. For
example, Sung et al. (2022) and Alayrac et al. (2022) insert into the PLMs new
layers that are trained from scratch. Tsimpoukelli et al. (2021) train vision
encoders that output soft prompts to frozen PLMs. Chen et al. (2022) and
Eichenberg et al. (2021) train both the vision encoders and new layers
inserted into PLMs. In the zero-shot setting, various vision-language
pretraining objectives are employed, such as image captioning Alayrac et al.
(2022) and image-conditioned masked language modeling Jin et al. (2022).

From the perspective of general-purpose AI, the ability to perform new tasks
by simply recombining large-scale pretrained models, or foundation models
Bommasani et al. (2021), without architectural changes or extra training would
be highly desirable. Such a system would be able to dynamically adjust to
previously unknown tasks by simply rewiring a small number of foundation
models. However, to obtain high performance without some form of end-to-end
training would seem difficult, if not impossible.

We present Plug-and-Play VQA (PnP-VQA), a framework for zero-shot visual
question answering which conjoins large pretrained models with zero additional
training and achieves state-of-the-art performance on zero-shot VQAv2 Goyal et
al. (2017) and GQA Hudson and Manning (2019). For the purpose of bridging the
vision and language modalities, we employ a pretrained vision-language model
(PVLM) Li et al. (2022b) that describes visual information with textual
captions. In order to obtain relevant and informative captions, we apply a
network interpretability technique Selvaraju et al. (2017) to detect image
patches that are relevant to the question. After that, we generate captions
stochastically for these image patches. Finally, we employ a PLM Khashabi et
al. (2022) to answer the question from the captions.

Research in cognitive science and neuroscience suggests that the human
cognitive system is largely modular Shettleworth (2012); Bertolero et al.
(2015). For instance, the pioneering work of Fodor (1983) argued that the low-
level human cognition is constituted of several fast, autonomous, and domain-
specific modules. For purely practical purposes, a modular design of
artificial general intelligence would make it easy to harness rapid progress
in each individual component, as the components can be individually replaced
and updated without affecting other parts of the system. With this paper, we
offer such a modular design for zero-shot VQA that leverages recent advances
in PLM and PVLMs and combines them with an innovative application of network
interpretability.

We summarize our contributions as follows:

  * •

We introduce PnP-VQA, a modular framework for zero-shot VQA without training.
Its flexibility allows PnP-VQA to jointly evolve as pretrained models continue
to advance.

  * •

Besides natural language, we propose the use of network interpretation as the
interface between pretrained LMs and VLMs. With an interpretability technique,
we create image captions that extensively cover information relevant to the
question, which enable accurate QA.

  * •

We demonstrate state-of-the-art zero-shot VQA performance on multiple
benchmarks. On VQAv2, PnP-VQA11B obtains 8.5% improvement over Flamingo80B
Alayrac et al. (2022), which applies extensive end-to-end VL-pretraining. On
GQA, PnP-VQAlarge outperforms FewVLMlarge Jin et al. (2022) by 9.1%.

##  2 Related Work

Figure 1:  The system architecture of PnP-VQA, consisting of three pretrained
modules: (1) an image-question matching module that identifies image patches
relevant to the question, (2) an image captioning module that generates a
diverse set of captions, (3) a question answering module that generates an
answer given the question and captions. For the image-question matching module
and image captioning module, we adopt BLIP Li et al. (2022b). For the question
answering module, we adopt UnifiedQAv2 Khashabi et al. (2022).

Large-scale image-text pretraining of neural networks is a popular research
direction. Various vision-language pretraining tasks have been proposed,
including image-conditioned language modeling Tsimpoukelli et al. (2021);
Alayrac et al. (2022), masked language modeling Tan and Bansal (2019); Lu et
al. (2019); Li et al. (2021b), prefix language modeling Wang et al. (2022),
image-text matching Li et al. (2019); Chen et al. (2020); Li et al. (2020) and
image-text contrastive learning Radford et al. (2021); Jia et al. (2021); Li
et al. (2021a). After pretraining, several models exhibit zero-shot
capabilities in image-text retrieval Jia et al. (2021); Radford et al. (2021);
Zeng et al. (2022b) and image captioning Wang et al. (2022); Li et al.
(2022b). However, zero-shot VQA remains a challenging task due to its high
requirement on the model’s reasoning ability.

Adapting PLMs for zero-shot VQA has shown promising results. In order to
incorporate vision information into PLMs, most existing methods perform
additional vision-language training on image-text data. Frozen Tsimpoukelli et
al. (2021) trains the vision encoder while keeping the gigantic PLM frozen to
retain its knowledge in question answering. The output from the vision encoder
is prepended to the text as prompts to the frozen language model. FewVLM Jin
et al. (2022) finetunes the PLM using the prefix language modeling and masked
language modeling objectives. VLKD Dai et al. (2022) distills multimodal
knowledge to PLM by using CLIP Radford et al. (2021) as the teacher model
during finetuning. Flamingo Alayrac et al. (2022) adds additional layers to
both the pretrained vision model and the PLM and trains the new layers on
billions of image-text pairs.

Different from the above work, PnP-VQA directly employs pretrained models with
neither architectural modifications nor additional training.

Most similar to our work, PICa Yang et al. (2022) converts an image to a
single caption and adopts GPT-3 Brown et al. (2020) for zero-shot VQA. In
comparison, PnP-VQA generates multiple question-guided captions and performs
fusion of captions after encoding to effectively utilize a large number of
captions, yielding considerable performance gains.

An orthogonal research direction for zero-shot VQA is to train the VLMs on
synthetic VQA examples generated from captions Changpinyo et al. (2022);
Banerjee et al. (2021). PnP-VQA does not require additional training.

Natural language as an intermediate representation or interface between
different models or multiple steps of reasoning is an emerging machine
learning strategy. It dates back to at least Andreas et al. (2018) and saw
renewed interest in the past few months due to the prevalence of large PLMs.
Andreas et al. (2018) and Vong and Lake (2022) learn natural language
descriptions that function as few-shot classifiers within an image-text
matching model. Bostrom et al. (2022) generate intermediate reasoning steps
with finetuned PLMs. Zhou et al. (2022) prompt a PLM to generate subproblem
descriptions for a complex problem, and feed the subproblems back to the PLM
to solve hierarchically. Wu et al. (2022) chain PLM outputs and inputs. Zeng
et al. (2022a) show that language-conjoined LM and VLM successfully perform
captioning and retrieval but do not evaluate their models on VQA. In
comparison, PnP-VQA adopts both natural language and network interpretation as
the interface between different pretrained models.

##  3 Method

The central idea of Plug-and-Play VQA (PnP-VQA) is to establish an interface
between a pretrained language model and a pretrained vision-language model
without training. We demonstrate that natural language image captions and
network saliency maps together serve as an effective interface. Ideally, the
generated captions should thoroughly cover information that is present in the
image and be relevant to the question. We foster relevance by identifying
image patches most related to the question with a saliency map-based
interpretability technique and generating captions from these patches only.
Further, we promote coverage by injecting stochasticity, including random
sampling of relevant image patches and of the textual tokens during caption
generation.

The overall system architecture (Figure 1) consists of three modules:

  1. 1.

an image-question matching module that identifies the relevant image patches
given a question,

  2. 2.

an image captioning module that generates a diverse set of captions from a set
of image patches, and

  3. 3.

a question answering module that outputs an answer given the question and the
generated captions.

In this section, we introduce the three modules in detail.

###  3.1 Matching Image Patches and Questions

An image serves as a rich source of information, but the question at hand is
likely focused only on particular objects or regions. Therefore, we encourage
PnP-VQA to generate captions that describe image regions relevant to the
question instead of generic captions with no specific aim.

We accomplish this goal by leveraging BLIP Li et al. (2022b), a large-scale
pretrained vision-language model that contains a network branch outputting a
similarity score $\text{sim}(v,t)$ between an image $v$ and a text $t$. This
branch, called Image-grounded Text Encoder (ITE), employs a vision transformer
Dosovitskiy et al. (2021) that encodes the image, and a textual encoder that
attends to the image features using cross-attention. As input to the image
encoder, the image is equally divided into $K$ patches.

Figure 2:  Examples of generic captions (from all patches) based on the
original image and question-guided captions (from the sampled patches) based
on the GradCAM heatmaps on VQAv2 data. For illustrative purposes, we highlight
words in green to indicate correct answer predictions and the cues from
captions. Words in red indicate wrong answer predictions.

To identify relevant image patches, we feed the image $v$ and the question $t$
to the ITE network and apply a variation of GradCAM Selvaraju et al. (2017), a
feature-attribution interpretability technique, that aggregates all cross-
attention maps using weights from the gradients. Formally, let us denote image
patch features as $X\in\mathbb{R}^{K\times D_{v}}$, where $K$ is the number of
image patches and $D_{v}$ the image feature dimension. We denote textual
features as $Y\in\mathbb{R}^{M\times D_{t}}$, where $M$ is the number of
textual tokens and $D_{t}$ the text feature dimension. For every cross-
attention head, we have parameter matrices $W_{Q}\in\mathbb{R}^{D_{t}\times
D_{t}}$ and $W_{K}\in\mathbb{R}^{D_{v}\times D_{t}}$ . The cross-attention
scores, $A\in\mathbb{R}^{M\times K}$, can be written as

$A=\operatorname{softmax}\left(\frac{YW_{Q}W_{K}^{\top}X^{\top}}{\sqrt{D_{t}}}\right).$
(1)

The $j$th row of $A$ indicates the amount of attention the $j$th textual token
allocates to all image patches. At a selected layer of the ITE network, we
compute the derivative of the similarity score w.r.t the cross-attention
score, $\partial\text{ sim}(v,t)/\partial A$, and multiply the gradient matrix
element-wise with the cross-attention scores. The relevance of the $i$th image
patch, $\text{rel}(i)$, takes the average over $H$ attention heads and the sum
over $M$ textual tokens:

$\text{rel}(i)=\frac{1}{H}\sum^{M}_{j=1}\sum^{H}_{h=1}\max\left(0,\frac{\partial\text{
sim}(v,t)}{\partial A_{ji}^{(h)}}\right)A_{ji}^{(h)},$ (2)

where the superscript (h) denotes the index of attention heads. For every
caption we generate, we sample a subset of $K^{\prime}$ image patches with
probability proportional to the patch relevance. The captioning module sees
the sampled patches only.

We provide the following motivation for the technique. The attention matrix
$A$ may be taken as indicative of patch importance. However, much redundancy
exists among these matrices and many attention heads may be pruned with little
performance loss Bian et al. (2021), suggesting that some scores are
uninformative. Inspired by GradCAM, we filter out uninformative attention
scores by multiplication with the gradient which could cause an increase in
the image-text similarity.

Image Patch Sampling Strategy | Num. of Captions | VQAv2 | OK-VQA | GQA
---|---|---|---|---
No captions | 0 | 33.4 | 10.3 | 25.9
All patches (generic captions) | 5 | 53.5 | 26.6 | 36.5
Uniform random sampling | 5 | 52.0 | 25.5 | 36.2
Question-guided patch sampling | 5 | 56.3 | 27.0 | 37.9
Human-written captions from MS COCO | 5 | 56.9 | 28.1 | -
All patches (generic captions) | 100 | 58.6 | 31.9 | 39.8
Uniform random sampling | 100 | 58.4 | 32.4 | 40.4
Question-guided patch sampling | 100 | 62.1 | 34.1 | 42.3

Table 1:  Comparison of different sampling strategies for image patches. 100
question-guided captions surpass the performance of 5 human-written captions
from MS COCO.  Figure 3:  Two methods to process multiple captions with a
question answering model. (a) Fusion-in-Encoder (FiE), which concatenates the
captions as a long input paragraph to the encoder. (b) Fusion-in-Decoder
(FiD), which encodes each caption with the question individually and
concatenates all encoded representations as input to the cross-attention
mechanism of the decoder.

Figure 2 shows some examples of generic captions and question-guided captions
with associated relevance heatmaps. We can clearly observe that question-
guided captions contain more relevant information that helps produce the
correct answers.

Table 1 gives a quantitative analysis about the effect of different patch
selection methods on zero-shot VQA performance across three datasets.
Question-guided patch sampling substantially outperforms generic captioning
using all patches and random patch sampling, especially when the number of
captions is large. 100 question-guided captions outperform the 5 human-written
captions from MS COCO by 5.2% on VQAv2 and 6.0% on OK-VQA, demonstrating the
merit of the proposed approach.

###  3.2 Informative Image Captioning

Even with relevant image regions, there may still be more than one way to
describe these regions. Some descriptions may contain the desired answer to
the question, whereas others may not. Without the ability to identify the
answer _a priori_ , we aim to generate maximally diverse captions to provide
coverage of possible answers.

We adopt the image captioning network branch from BLIP Li et al. (2022b) and
apply stochastic top-$k$ sampling Fan et al. (2018) instead of beam search,
which is known to produce dull and repetitive captions Vijayakumar et al.
(2018); Holtzman et al. (2020). The input to the network contains the
$K^{\prime}$ image patches sampled according to relevance (see §3.1). We
prepend a short prompt, “a picture of ” as input to the text decoder. We
repeat this process to generate $N$ captions per image to encourage diversity
of captions and coverage of visual content. To prevent repetition, we keep a
generated caption only if it is not subsumed by any previous caption as an
exact substring.

###  3.3 Answering the Question

Figure 4:  Comparison between Fusion-in-Encoder and Fusion-in-Decoder for
VQAv2, OK-VQA and GQA.

The question-answering encoder-decoder model is pretrained on text data only
and can only process text. Therefore, we include the question and the
generated captions as input to the model. As discussed in §3.2, the image
captioning module generates multiple diverse captions. To process such long
inputs efficiently, we adopt the Fusion-in-Decoder (FiD) strategy Izacard and
Grave (2021).

We illustrate the FiD strategy in Figure 3 by comparing it with the more
straightforward Fusion-in-Encoder (FiE), which concatenates the question and
all captions into a long paragraph as input to the encoder. In contrast, FiD
encodes each caption with the question separately and concatenates the
_encoded representations_ of all tokens from all captions. The result is fed
as input to the decoder and is processed through the cross-attention
mechanism. Since the time complexity of the self-attention mechanism scales
quadratically with input length, whereas the cross-attention scales linearly
with the encoder’s output length, FiD is much more efficient than FiE.
Further, FiE is constrained by the maximum input length of the encoder, caused
by the positional encoding, but FiD does not have this constraint. Hence, with
FiD, PnP-VQA can benefit from even more captions.

We plot the performance of FiD and FiE against the number of captions in
Figure 4. Initially, both methods improve as the number of captions increases.
However, the performance of FiE is capped at around 40 captions when the
maximum input length is exceeded, whereas the performance of FiD continues to
rise.

##  4 Experiments

###  4.1 Datasets and Evaluation

We adopt multiple zero-shot VQA benchmarks, including the validation set
(214,354 questions) and test-dev set (107,394 questions) of VQAv2 Goyal et al.
(2017), the test set (5,046 questions) of OK-VQA Marino et al. (2019), and the
test-dev set (12,578 questions) of GQA-balanced Hudson and Manning (2019). We
include the VQAv2 validation set as a few recent works Tsimpoukelli et al.
(2021); Jin et al. (2022) evaluate their performance on this dataset only. We
obtain the answer by open-ended generation and perform evaluation based on
exact matching. We report soft-accuracy Goyal et al. (2017) for VQAv2 and OK-
VQA to account for multiple ground truth answer; for GQA, we report the
standard accuracy.

###  4.2 Implementation Details

To obtain the image-question matching module and image captioning module, we
adopt BLIP Li et al. (2022b) with the ViT-L/16 architecture pretrained on 129M
image-text pairs. The original BLIP-ITM and BLIP-Caption models further
finetune on the 2017 train split of COCO Captions Lin et al. (2014), which
partially overlaps with VQAv2 and OKVQA. To prevent data leak, we instead
finetune on the 2014 train split of COCO Captions, which does not overlap with
the VQA evaluation datasets. We emphasize that this represents less, not more,
training compared to the publicly released BLIP.

For the question answering module, we adopt UnifiedQAv2 Khashabi et al. (2022)
trained on diverse textual QA datasets. It is worth noting that UnifiedQAv2 is
completely unaware of the visual modality during training. Therefore, its
training data do not overlap with the VQA datasets.

Unless otherwise stated, we utilize a total of 100 captions per question. We
select the 8th cross-attention layer of the ITE network for GradCAM. We sample
$K^{\prime}=20$ image patches for the generation of each caption, and use
$k=50$ for top-$k$ decoding (see Fig. 9 in Appendix B). For VQAv2 and OK-VQA,
we apply FiD and encode the question with one caption at a time. However, for
GQA, we encode each question with a group of 5 captions. GQA requires
compositional visual reasoning and thus benefits from more contextual
information per question. We perform experiments using LAVIS Li et al. (2022a)
on 8 Nvidia A100 GPUs.

Method | Language | Vision | VQAv2 | OK-VQA | GQA
---|---|---|---|---|---
Model | #Params | VL-aware | Model | #Params | VL-aware | Val | Test-dev | Test | Test-dev
_Pretrained models conjoined by end-to-end VL training._
VL-T5no-vqa | T5 | 224M | ✓ | Faster R-CNN | 64M | ✗ | 13.5 | - | 5.8 | 6.3
FewVLMbase | T5 | 224M | ✓ | Faster R-CNN | 64M | ✗ | 43.4 | - | 11.6 | 27.0
FewVLMlarge | T5 | 740M | ✓ | Faster R-CNN | 64M | ✗ | 47.7 | - | 16.5 | 29.3
VLKDViT-B/16 | BART | 407M | ✓ | ViT-B/16 | 87M | ✓ | 38.6 | 39.7 | 10.5 | -
VLKDViT-L/14 | BART | 408M | ✓ | ViT-L/14 | 305M | ✓ | 42.6 | 44.5 | 13.3 | -
Flamingo3B | Chinchilla-like | 2.6B | ✓ | NFNet-F6 | 629M | ✓ | - | 49.2 | 41.2 | -
Flamingo9B | Chinchilla-like | 8.7B | ✓ | NFNet-F6 | 629M | ✓ | - | 51.8 | 44.7 | -
Flamingo80B | Chinchilla | 80B | ✓ | NFNet-F6 | 629M | ✓ | - | 56.3 | 50.6 | -
Frozen | GPT-like | 7B | ✗ | NF-ResNet-50 | 40M | ✓ | 29.5 | - | 5.9 | -
_Pretrained models conjoined by natural language and zero training._
PICa | GPT-3 | 175B | ✗ | VinVL-Caption | 259M | ✓ | - | - | 17.7 | -
PnP-VQAbase | UnifiedQAv2 | 223M | ✗ | BLIP-Caption | 446M | ✓ | 54.3 | 55.2 | 23.0 | 34.6
PnP-VQAlarge | UnifiedQAv2 | 738M | ✗ | BLIP-Caption | 446M | ✓ | 57.5 | 58.8 | 27.1 | 38.4
PnP-VQA3B | UnifiedQAv2 | 2.9B | ✗ | BLIP-Caption | 446M | ✓ | 62.1 | 63.5 | 34.1 | 42.3
PnP-VQA11B | UnifiedQAv2 | 11.3B | ✗ | BLIP-Caption | 446M | ✓ | 63.3 | 64.8 | 35.9 | 41.9

Table 2:  Comparison with state-of-the-art models on zero-shot VQA. Flamingo
Alayrac et al. (2022) inserts additional parameters into the language model
and perform training using billion-scale vision-language data. The best
accuracy is bolded and the second best is underlined.

###  4.3 Comparison with State of the Arts

We compare with state-of-the-art methods that formulate zero-shot VQA as open-
ended answer generation. We categorize the methods based on how the pretrained
networks are conjoined. In the first group, including VL-T5no-vqa Cho et al.
(2021), FewVLM Jin et al. (2022), VLKD Dai et al. (2022), Flamingo Alayrac et
al. (2022), and Frozen Tsimpoukelli et al. (2021), a vision encoder (VE)
embeds the image as a dense matrix and feeds it to the pretrained language
model (PLM). After that, the system performs a round of end-to-end vision-
language (VL) training on tasks other than VQA, such as image captioning.
VL-T5no-vqa and FewVLM freeze the VE and finetune the PLM, whereas Frozen
freezes the PLM and trains the VE. VLKD finetunes both the PLM and part of VE.
Flamingo partially finetunes both the VE and the PLM. In the second group, the
two foundation models are not jointly trained. Instead, they use language in
the form of captions as the intermediate representation for an image. This
group includes PICa Yang et al. (2022) and our proposed model, PnP-VQA.

Table 2 shows the results. PnP-VQA outperforms previous methods by large
margins on VQAv2 and GQA. On VQAv2 test-dev, PnP-VQA11B outperforms the second
best technique, Flamingo80B Alayrac et al. (2022), by 8.5%. PnP-VQA3B
outperforms Flamingo80B by 7.2% despite its significantly smaller size and the
similar-sized Flamingo3B by 14.3%. On GQA, PnP-VQAlarge outperforms the
FewVLMlarge by 9.1%, with similar-sized PLM despite the lack of end-to-end
training. Only on OK-VQA, Flamingo performs better than PnP-VQA. OK-VQA
requires external knowledge not existing in the images and cannot be solved by
good captions alone. We hypothesize that the end-to-end training on the
gigantic vision-language dataset of Flamingo induces a mapping between images
and knowledge concepts that helps with OK-VQA. However, PnP-VQA is still
better on OK-VQA than all other baselines that not trained on the gigantic
Flamingo data. Compared with language-conjoined PICa Yang et al. (2022) with
175B parameters, PnP-VQA11B achieves a sizable improvement of 18.2%.

The results underscore the difficulty of zero-shot VQA using language models
without any vision-language (VL) training. PICa, with its 175B-parameter
language model, achieves comparable performance as FewVLMlarge, whose language
model is 236x smaller but finetuned on VL data. On the other hand, finetuning
the billion-scale language model could incur heavy computational cost and risk
catastrophic forgetting Tsimpoukelli et al. (2021); Alayrac et al. (2022).
PnP-VQA demonstrates the feasibility of a different paradigm: using billion-
scale pretrained language models for VQA with zero training.

##  5 Analysis

###  5.1 Are PnP-VQA captions informative?

Figure 5:  Analysis on the relationships between answer hit rate (AHR), VQA
accuracy, and the number of captions per question (N). (a) shows a positive
correlation between AHR and VQA accuracy. (b) shows the AHR increases with N,
where the proposed question-guided patch sampling produces captions with the
highest AHR.

Intuitively, if the captions contain the correct answer, the QA model would
have a higher chance to answer correctly. To measure the utility of captions,
we compute the answer hit rate (AHR), or the proportion of questions for which
at least one caption contains the ground-truth answer verbatim. Here we
exclude questions with yes/no answers as the meaning of “yes” and “no” can be
contextual and these two words appear rarely in captions.

Decoding Method | VQAv2 | OK-VQA | GQA
---|---|---|---
Beam search | 55.3 | 26.8 | 37.2
Temperature ($t$=0.5) | 61.2 | 32.1 | 41.4
Temperature ($t$=1) | 60.0 | 31.6 | 41.6
Nucleus ($p$=0.9) | 61.3 | 32.9 | 41.7
Nucleus ($p$=0.95) | 60.7 | 32.2 | 41.9
Top-$k$ ($k$=50) | 62.1 | 34.1 | 42.3
Top-$k$ ($k$=100) | 61.9 | 34.0 | 42.3

Table 3:  Ablation study on different caption decoding methods. PnP-VQA3B
performs well across the stochastic methods.

Figure 5(a) shows the correlation between the AHR and VQA accuracy, computed
over the VQAv2 validation set, for three techniques of image patch sampling:
question-guided sampling, uniform random sampling, and all patches. We observe
that, within each sampling method, the VQA accuracy increases as the AHR
increases. This corroborates our hypothesis that the presence of the answer in
the captions facilitates the generation of the correct answer.

The correlation between performance and AHR is not perfect, as AHR does not
capture other factors that may affect the answer accuracy, such as the
position of the answer in the sentence and the number of its occurrence.
However, AHR provides an easy-to-compute and useful measure for the
information quality of the captions.

Figure 5(b) shows how AHR changes with the number of captions. Among the three
techniques, question-guided sampling produces captions with the highest AHR.
Thus, we may attribute the good performance of PnP-VQA partially to its
informative, question-guided captions that directly contain the correct
answer. Further, as the number of captions increases from 20 to 100, question-
guided AHR increases from 71.8% to 84.0%. This demonstrates the benefit of
Fusion-in-Decoder, which allows PnP-VQA to utilize up to 100 captions.

###  5.2 How sensitive is PnP-VQA to the caption decoding method?

QA Model | #Params | VQAv2 | OK-VQA | GQA
---|---|---|---|---
GPT-J | 6B | 28.7 | 14.5 | 18.5
T0 | 3B | 49.6 | 26.6 | 32.3
T0 | 11B | 47.3 | 30.5 | 33.4
UnifiedQAv2 | 3B | 62.1 | 34.1 | 42.3
UnifiedQAv2 | 11B | 63.3 | 35.9 | 41.9

Table 4:  Ablation study on various textual question answering module for PnP-
VQA on zero-shot VQA. UnifiedQAv2 is a task-specific model pretrained for
question answering.

As the content of captions plays a crucial role in the performance of PnP-VQA,
we investigate the sensitivity to the choice of the caption decoding methods.
We test four methods, including the deterministic beam search and three
stochastic methods — temperature sampling Ficler and Goldberg (2017); Caccia
et al. (2020), nucleus sampling Holtzman et al. (2020), and top-$k$ sampling
Fan et al. (2018). We generate 100 captions from each method, and report the
results in Table 3. PnP-VQA performs very similarly across stochastic decoding
methods, but beam search results in a noticeable drop. Upon close inspection,
we observe that beam search generates repetitive captions that do not
sufficiently cover different aspects of the image.

###  5.3 Can PnP-VQA work with other textual QA models?

We experiment with two other PLMs as the question answering module for PnP-
VQA: T0 Sanh et al. (2022) and GPT-J Wang and Komatsuzaki (2021). T0 is an
encoder-decoder model which is pretrained in a multi-task fashion on a
collection of NLP tasks, including question answering. GPT-J is a decoder-only
model, a much smaller open-source alternative to GPT-3 Brown et al. (2020),
which is pretrained with a task-agnostic language modeling loss on a large-
scale text corpus. Table 4 shows that UnifiedQAv2 performs better on VQA tasks
compared to T0 and GPT-J. We attribute UnifiedQAv2’s good performance to the
fact that it is a task-specific question answering model with superior textual
QA performance. The result indicates that the choice of PLM is important when
performing zero-shot VQA with zero training. The modular and flexible design
of PnP-VQA leaves room for further performance improvements as more advanced
PLMs emerge.

##  6 Conclusion

We propose PnP-VQA, a framework with zero additional training for zero-shot
VQA by conjoining off-the-shelf pretrained models. PnP-VQA leverages an image-
question matching module to determine image patches relevant to the current
question. An image captioning module then generates question-guided captions,
which are processed by a question answering module to produce an answer. PnP-
VQA achieves state-of-the-arts performance on multiple VQA benchmarks. We hope
that our work will bring inspiration for further research in flexible, modular
AI systems for solving vision-language tasks.

##  7 Limitations

Like two sides of the same coin, the strengths and weaknesses of PnP-VQA both
result from the zero-training modular system design. PnP-VQA enjoys the power
of pretrained models but also inherits the bias from these models. It enjoys
the efficiency of zero training, but introduces additional inference cost due
to the multi-step process. Nevertheless, we believe that the strengths of PnP-
VQA outweigh its limitations, and welcome further investigations to help
debias pretrained models and improve inference speed.

##  8 Acknowledgments

Anthony Meng Huat Tiong is supported by Salesforce and Singapore Economic
Development Board under the Industrial Postgraduate Programme. Boyang Li is
supported by the Nanyang Associate Professorship and the National Research
Foundation Fellowship (NRF-NRFF13-2021-0006), Singapore. Any opinions,
findings, conclusions, or recommendations expressed in this material are those
of the authors and do not reflect the views of the funding agencies.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022.  Flamingo: a visual language model for few-shot learning.  _arXiv Preprint 2204.14198_.
  * Andreas et al. (2018) Jacob Andreas, Dan Klein, and Sergey Levine. 2018.  Learning with latent language.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2166–2179, New Orleans, Louisiana. Association for Computational Linguistics.
  * Banerjee et al. (2021) Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2021.  WeaQA: Weak supervision via captions for visual question answering.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3420–3435, Online. Association for Computational Linguistics.
  * Bertolero et al. (2015) Maxwell A. Bertolero, B. T. Thomas Yeo, and Mark D’Esposito. 2015.  The modular and integrative functional architecture of the human brain.  _Proceedings of the National Academy of Sciences_ , 112(49):E6798–E6807.
  * Bian et al. (2021) Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. 2021.  On attention redundancy: A comprehensive study.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 930–945, Online. Association for Computational Linguistics.
  * Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 95–136, virtual+Dublin. Association for Computational Linguistics.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.  _arXiv Preprint 2108.07258_.
  * Bostrom et al. (2022) Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. 2022.  Natural language deduction through search over statement compositions.  _arXiv Preprint 2201.06028_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Caccia et al. (2020) Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2020.  Language gans falling short.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Changpinyo et al. (2022) Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022.  All you may need for vqa are image captions.  In _North American Chapter of the Association for Computational Linguistics_.
  * Chen et al. (2022) Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022.  VisualGPT: Data-efficient adaptation of pretrained language models for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18030–18040.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  UNITER: universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 1931–1942. PMLR.
  * Dai et al. (2022) Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022.  Enabling multimodal generation on CLIP via vision-language knowledge distillation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2383–2395, Dublin, Ireland. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Eichenberg et al. (2021) Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. 2021.  Magma – multimodal augmentation of generative models through adapter-based finetuning.  _arXiv Preprint 2112.05253_.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  * Ficler and Goldberg (2017) Jessica Ficler and Yoav Goldberg. 2017.  Controlling linguistic style aspects in neural language generation.  In _Proceedings of the Workshop on Stylistic Variation_ , pages 94–104, Copenhagen, Denmark. Association for Computational Linguistics.
  * Fodor (1983) Jerry Alan Fodor. 1983.  _The Modularity of Mind_.  MIT Press.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , pages 6904–6913.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019.  GQA: A new dataset for real-world visual reasoning and compositional question answering.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 6700–6709. Computer Vision Foundation / IEEE.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 4904–4916. PMLR.
  * Jin et al. (2022) Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. 2022.  A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2763–2775, Dublin, Ireland. Association for Computational Linguistics.
  * Khashabi et al. (2022) Daniel Khashabi, Yeganeh Kordi, and Hannaneh Hajishirzi. 2022.  UnifiedQA-v2: Stronger generalization via broader cross-format training.  _arXiv preprint arXiv:2202.12359_.
  * Li et al. (2022a) Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. 2022a.  Lavis: A library for language-vision intelligence.
  * Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b.  BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_.
  * Li et al. (2021a) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021a.  Align before fuse: Vision and language representation learning with momentum distillation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 9694–9705. Curran Associates, Inc.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  VisualBERT: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. (2021b) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021b.  UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2592–2607, Online. Association for Computational Linguistics.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX_ , volume 12375 of _Lecture Notes in Computer Science_ , pages 121–137. Springer.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: common objects in context.  In _Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V_ , volume 8693 of _Lecture Notes in Computer Science_ , pages 740–755. Springer.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  OK-VQA: a visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3195–3204.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 618–626.
  * Shettleworth (2012) Sara J Shettleworth. 2012.  Modularity, comparative cognition and human uniqueness.  _Philosophical transactions of the Royal Society of London. Series B, Biological sciences_ , 367(1603):2794–2802.
  * Sung et al. (2022) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022.  VL-Adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5227–5237.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111, Hong Kong, China. Association for Computational Linguistics.
  * Tsimpoukelli et al. (2021) Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021.  Multimodal few-shot learning with frozen language models.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 200–212. Curran Associates, Inc.
  * Vijayakumar et al. (2018) Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018.  Diverse beam search: Decoding diverse solutions from neural sequence models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Vong and Lake (2022) Wai Keen Vong and Brenden M. Lake. 2022.  Few-shot image classification by generating natural language rules.  In _ACL Workshop on Learning with Natural Language Supervision_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2022) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022\.  SimVLM: Simple visual language model pretraining with weak supervision.  In _International Conference on Learning Representations, ICLR 2022_. OpenReview.net.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv Preprint 2109.01652_.
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022.  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts.  In _CHI Conference on Human Factors in Computing Systems_ , CHI ’22, New York, NY, USA. Association for Computing Machinery.
  * Yang et al. (2022) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022.  An empirical study of GPT-3 for few-shot knowledge-based VQA.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Zeng et al. (2022a) Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. 2022a.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_.
  * Zeng et al. (2022b) Yan Zeng, Xinsong Zhang, and Hang Li. 2022b.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 25994–26009. PMLR.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv Preprint 2205.10625_.

##  Appendix A Visualization

In the appendix, we show visualizations of GradCAM heatmaps and the generated
captions for VQAv2, OK-VQA, and GQA in following pages.

Figure 6:  Examples from VQAv2. We show generic captions (from all patches)
based on the original image and question-guided captions (from the sampled
patches) based on the GradCAM heatmaps. For illustrative purposes, we
highlight words in green to indicate correct answer predictions and the cues
in captions. Words in red indicate wrong answer predictions.  Figure 7:
Examples from OK-VQA. We show generic captions (from all patches) based on the
original image and question-guided captions (from the sampled patches) based
on the GradCAM heatmaps. For illustrative purposes, we highlight words in
green to indicate correct answer predictions and the cues in captions. Words
in red indicate wrong answer predictions.  Figure 8:  Examples from GQA. We
show generic captions (from all patches) based on the original image and
question-guided captions (from the sampled patches) based on the GradCAM
heatmaps. For illustrative purposes, we highlight words in green to indicate
correct answer predictions and the cues in captions. Words in red indicate
wrong answer predictions.  Figure 9:  VQAv2 validation set accuracy using (a)
different cross-attention layer on which GradCAM is computed using
$K^{\prime}=50$. (b) different number of image patches sampled for caption
generation using GradCAM computed at 8th cross-attention layer.

##  Appendix B Hyperparameter sensitivity

We study how VQAv2 validation accuracy varies with different cross-attention
layer used for GradCAM and number of image patches sampled for question-guided
caption generation. Figure 9(a) shows no clear relationship between VQA
accuracy and the cross-attention layer used for GradCAM. The maximum
difference in VQA accuracy across different cross-attention layers is 3%.
Figure 9(b) shows that VQA accuracy has a negative correlation with the number
of sampled image patches. As $K^{\prime}$ increases, the sampled patches
become less relevant to the questions, and question-guided patch sampling
becomes akin to using all patches.