# Large Multimodal Agents: A Survey

Junlin Xie♣♡∗ Zhihong Chen♣♡ Ruifei Zhang♣♡ Xiang Wan♣ Guanbin Li♠

♡The Chinese University of Hong Kong, Shenzhen
♣Shenzhen Research Institute of Big Data, ♠Sun Yat-sen University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contributionCorresponding
author

###### Abstract

Large language models (LLMs) have achieved superior performance in powering
text-based AI agents, endowing them with decision-making and reasoning
abilities akin to humans. Concurrently, there is an emerging research trend
focused on extending these LLM-powered AI agents into the multimodal domain.
This extension enables AI agents to interpret and respond to diverse
multimodal user queries, thereby handling more intricate and nuanced tasks. In
this paper, we conduct a systematic review of LLM-driven multimodal agents,
which we refer to as large multimodal agents (LMAs for short). First, we
introduce the essential components involved in developing LMAs and categorize
the current body of research into four distinct types. Subsequently, we review
the collaborative frameworks integrating multiple LMAs, enhancing collective
efficacy. One of the critical challenges in this field is the diverse
evaluation methods used across existing studies, hindering effective
comparison among different LMAs. Therefore, we compile these evaluation
methodologies and establish a comprehensive framework to bridge the gaps. This
framework aims to standardize evaluations, facilitating more meaningful
comparisons. Concluding our review, we highlight the extensive applications of
LMAs and propose possible future research directions. Our discussion aims to
provide valuable insights and guidelines for future research in this rapidly
evolving field. An up-to-date resource list is available at
https://github.com/jun0wanan/awesome-large-multimodal-agents.

##  1 Introduction

Figure 1: Representative research papers from top AI conferences on LLM-
powered multimodal agents, published between November 2022 and February 2024,
are categorized by model names, with earlier publication dates corresponding
to names listed earlier.

An agent is a system capable of perceiving its environment and making
decisions based on these perceptions to achieve specific goals [56]. While
proficient in narrow domains, early agents[35, 50] often lack adaptability and
generalization, highlighting a significant disparity with human intelligence.
Recent advancements in large language models (LLMs) have begun to bridge this
gap, where LLMs enhance their capabilities in command interpretation,
knowledge assimilation [36, 78], and mimicry of human reasoning and learning
[21, 66]. These agents use LLMs as their primary decision-making tool and are
further enhanced with critical human-like features, such as memory. This
enhancement allows them to handle a variety of natural language processing
tasks and interact with the environment using language [40, 38].

However, real-world scenarios often involve information that spans beyond
text, encompassing multiple modalities, with a significant emphasis on the
visual aspect. Consequently, the next evolutionary step for LLM-powered
intelligent agents is to acquire the capability to process and generate
multimodal information, particularly visual data. This ability is essential
for these agents to evolve into more robust AI entities, mirroring human-level
intelligence. Agents equipped with this capability are referred to as large
multimodal agents (LMAs) in our paper.111The name is inherited from Large
Multimodal Models (LMMs). Typically, they face more sophisticated challenges
than language-only agents. Take web searching for example, an LMA first
requires the input of user requirements to look up relevant information
through a search bar. Subsequently, it navigates to web pages through mouse
clicks and scrolls to browse real-time web page content. Lastly, the LMA needs
to process multimodal data (e.g., text, videos, and images) and perform multi-
step reasoning, including extracting key information from web articles, video
reports, and social media updates, and integrating this information to respond
to the user’s query. We note that existing studies in LMAs were conducted in
isolation therefore it is necessary to further advance the field by
summarizing and comparing existing frameworks. There exist several surveys
related to LLM-powered agents [60, 42, 49] while few of them focused on the
multimodal aspects.

In this paper, we aim to fill the gap by summarizing the main developments of
LMAs. First, we give an introduction about the core components (§2) and
propose a new taxonomy for existing studies (§3) with further discussion on
existing collaborative frameworks (§4). Regarding the evaluation, we outline
the existing methodologies for assessing the performance of LMAs, followed by
a comprehensive summary (§5). Then, the application section provides an
exhaustive overview of the broad real-world applications of multimodal agents
and their related tasks (§6). We conclude this work by discussing and
suggesting possible future directions for LMAs to provide useful research
guidance.

##  2 The Core Components of LMAs

Type Model Task Focus Planner Action Multi- Long Text Image Video Audio Model
Format Inspect Planning Method Action Type Action Learning Agent Memory Type I
VisProg [11] ✓ ✓ ✗ ✗ GPT3.5 Program ✗ Fix T (VFMs & Python) Prompt ✗ ✗
ControlLLM [11] ✓ ✓ ✗ ✗ GPT4 Language ✓ Fix T (VFMs & Python) Prompt ✗ ✗
Visual ChatGPT [57] ✗ ✓ ✗ ✗ GPT3.5 Language ✗ Fix T (VFMs) Prompt ✗ ✗ ViperGPT
[43] ✓ ✓ ✓ ✗ GPT3.5 Program ✗ Fix T (VFMs & API & Python) Prompt ✗ ✗ MM-ReAct
[70] ✗ ✓ ✓ ✗ ChatGPT & GPT3.5 Language ✓ Fix T (Web & API) Prompt ✗ ✗
Chameleon [30] ✓ ✓ ✗ ✗ GPT3.5 Language ✗ Fix T (VFMs & API & Python & Web)
Prompt ✗ ✗ HuggingGPT [41] ✗ ✓ ✓ ✗ GPT3.5 Language ✗ Fix T (VFMs) Prompt ✗ ✗
CLOVA [10] ✓ ✓ ✗ ✗ GPT4 Language ✓ Dynamic T (VFMs & API) Prompt ✗ ✗ CRAFT
[74] ✓ ✓ ✗ ✗ GPT4 Program ✗ Fix T (Custom tools) Prompt ✗ ✗ Cola [3] ✓ ✓ ✓ ✗
ChatGPT Language ✗ Fix T (VFMs) Prompt ✗ ✗ M3 [25] ✗ ✓ ✓ ✗ GPT3.5 Language ✓
Dynamic T (VFMs & API) Prompt ✗ ✗ DEPS [52] ✓ ✓ ✗ ✗ GPT-4 Language ✓ Dynamic E
Prompt ✗ ✗ GRID [45] ✓ ✓ ✗ ✗ GPT-4 Language ✓ Dynamic T (VFMs & API) Prompt ✗
✗ DroidBot-GPT [54] ✓ ✓ ✗ ✗ ChatGPT Language ✗ Dynamic V Prompt ✗ ✗ ASSISTGUI
[8] ✓ ✓ ✗ ✗ GPT-4 Language ✓ Dynamic V(GUI parser) Prompt ✗ ✗ GPT-Driver [32]
✓ ✓ ✗ ✗ GPT-3.5 Language ✗ Dynamic E Prompt ✗ ✗ LLaVA-Interactive [4] ✓ ✓ ✗ ✗
GPT-4 Language ✗ Dynamic T (VFMs) Prompt ✗ ✗ MusicAgent [73] ✓ ✗ ✗ ✓ ChatGPT
Language ✗ Dynamic T (Music-Models) Prompt ✗ ✗ AudioGPT [13] ✓ ✗ ✗ ✓ GPT-4(V)
Language ✗ Fix T (API) Prompt ✗ ✗ AssistGPT [9] ✗ ✓ ✓ ✗ GPT3.5 Lang. & Prog. ✓
Dynamic T (VFMs & API) Prompt ✗ ✗ Mulan[20] ✓ ✓ ✗ ✗ GPT3.5 Language ✓ Dynamic
T (VFMs & Python) Prompt ✗ ✗ Mobile-Agent[48] ✓ ✓ ✗ ✗ GPT4 Language ✓ Dynamic
V & T(VFMs) Prompt ✗ ✗ Type II GPT-Driver [32] ✗ ✓ ✓ ✗ GPT4 Language ✗ Fix E
Learning ✗ ✗ LLAVA-PLUS [23] ✗ ✓ ✓ ✗ Llava Language ✓ Dynamic T (VFMs)
Learning ✗ ✗ GPT4tools [67] ✗ ✓ ✓ ✗ Llama Language ✓ Dynamic T (VFMs) Learning
✗ ✗ Tool-LMM [46] ✓ ✓ ✓ ✓ Vicuna Language ✗ Dynamic T (VFMs & API) Learning ✗
✗ STEVE [79] ✓ ✓ ✗ ✗ STEVE-13B Program ✗ Fix E Learning ✗ ✗ EMMA [68] ✓ ✓ ✗ ✗
LMDecoder Language ✗ Fix E Learning ✗ ✗ Auto-UI [75] ✓ ✓ ✗ ✗ LMDecoder
Language ✗ Dynamic E Learning ✗ ✗ WebWISE [44] ✓ ✓ ✗ ✗ LMDecoder Language ✓
Dynamic E Learning ✗ ✗ Type III DORAEMONGPT [71] ✓ ✗ ✓ ✗ GPT4 Language ✗
Dynamic T (VFMs) Prompt ✗ ✓ ChatVideo [47] ✓ ✓ ✓ ✗ ChatGPT Language ✗ Fix T
(VFMs) Prompt ✗ ✓ OS-Copilot [59] ✓ ✓ ✗ ✗ GPT4 Language ✓ Dynamic V Prompt ✗ ✓
Type IV Openagents [62] ✓ ✓ ✓ ✗ GPT3.5 & GPT4 Language ✓ Dynamic V & T Prompt
✗ ✓ MEIA [27] ✓ ✓ ✗ ✗ GPT3.5 & GPT4 Language ✗ Fix E Prompt ✗ ✓ JARVIS-1 [51]
✓ ✓ ✗ ✗ GPT-4 Language ✓ Dynamic E Prompt ✗ ✓ AppAgent [69] ✓ ✓ ✗ ✗ GPT-4(V)
Language ✓ Dynamic E Prompt ✗ ✓ MM-Navigator [64] ✓ ✓ ✗ ✗ GPT-4(V) Language ✗
Dynamic T (API) Prompt ✗ ✓ DLAH [7] ✓ ✓ ✗ ✗ GPT-3.5 Language ✗ Dynamic
V(Simulator-interfaces) Prompt ✗ ✓ Copilot [77] ✓ ✗ ✗ ✓ GPT-3.5 Language ✓
Dynamic T (Music-Models) Prompt ✗ ✓ WavJourney [26] ✓ ✗ ✗ ✓ GPT-4 Program ✗
Dynamic T (Music-Models) Prompt ✗ ✓ Multi-agent AVIS [12] ✗ ✓ ✓ ✗ GPT4
Language ✓ Dynamic T (VFMs) Prompt ✓ ✗ MP5 [37] ✓ ✓ ✗ ✗ GPT-4 Language ✓
Dynamic E Prompt ✓ ✓ MemoDroid [17] ✓ ✓ ✗ ✗ GPT-4 Language ✓ Dynamic V(VFMs &
API) Prompt ✓ ✓ DiscussNav [29] ✓ ✓ ✗ ✗ GPT-4 & ChatGPT Language ✗ Dynamic E
Prompt ✓ ✗

Table 1:  This presentation delineates the component details of all LMAs,
encompassing their task-specific modalities, the models outlined by planners,
the methodologies and formats employed in planning, the variety of actions
involved, the extent of multi-agent collaboration, and the incorporation of
long-term memory. Within this table, ‘V’ represents the virtual action, ‘T’
indicates the use of a tool, and ‘E’ embodies the physical action.

In this section, we detail four core elements of LMAs including perception,
planning, action, and memory.

#### Perception.

Perception is a complex cognitive process that enables humans to collect and
interpret environmental information. In LMAs, the perception component
primarily focuses on processing multimodal information from diverse
environments. As illustrated in Table 1, LMAs in different tasks involve
various modalities. They require extracting key information from these
different modalities that is most beneficial for task completion, thereby
facilitating more effective planning and execution of the tasks.

Early research [57, 43, 70, 9] on processing multimodal information often rely
on simple correlation models or tools to convert images or audio into text
descriptions. However, this conversion approach tends to generate a large
amount of irrelevant and redundant information, particularly for complex
modalities (e.g., video). Along with the input length constraint, LLMs
frequently face challenges in effectively extracting pertinent information for
planning. To address this issue, recent studies [71, 47] have introduced the
concept of sub-task tools, which are designed to handle sophisticated data
types. In an environment resembling the real world (i.e., open-world games),
[51] proposed a novel method for processing non-textual modal information.
This approach begins by extracting key visual vocabulary from the environment
and then employs the GPT model to further refine this vocabulary into a series
of descriptive sentences. As LLMs perceive visual modalities within the
environment, they use them to retrieve the most relevant descriptive
sentences, which effectively enhances their understanding of the surroundings.

Figure 2: Illustrations on four types of LMAs: (a) Type I: Closed-source LLMs
as Planners w/o Long-term Memory. They mainly use prompt techniques to guide
closed-source LLMs in decision-making and planning to complete tasks without
long memory. (b) Type II:Finetuned LLMs as Planners w/o Long-term Memory. They
use action-related data to finetune existing open-source large models,
enabling them to achieve decision-making, planning, and tool invocation
capabilities comparable to closed-source LLMs. Unlike (a) and (b), (c) and (d)
introduce long-term memory functions, further enhancing their generalization
and adaptation abilities in environments closer to the real world. However,
because their planners use different methods to retrieve memories, they can be
further divided into: (c) Type III: Planners with Indirect Long-term Memory;
(d) Type IV: Planners with Native Long-term Memory. Figure 3: Illustrations on
two types of multi-agent frameworks: in these two frameworks, facing tasks or
instructions from the environment, the completion relies on the cooperation of
multiple agents. Each agent is responsible for a specific duty, which may
involve processing environmental information or handling decision-making and
planning, thus distributing the pressure that would otherwise be borne by a
single- agent to complete the task. The unique aspect of framework (b) is its
long-term memory capability.

#### Planning.

Planners play a central role in LMAs, akin to the function of the human brain.
They are responsible for deep reasoning about the current task and formulating
corresponding plans. Compared to language-only agents, LMAs operate in a more
complicated environment, making it more challenging to devise reasonable
plans. We detail planners from four perspectives (models, format, inspection &
reflection, and planning methods):

  * •

Models: As shown in Table 1, existing studies employ different models as
planners. Among them, the most popular ones are GPT-3.5 or GPT-4 [43, 41, 9,
30, 57, 51]. Yet, these models are not publicly available and therefore some
studies have begun shifting towards using open-source models, such as LLaMA
[67] and LLaVA [23], where the latter can directly process information of
multiple modalities, enhancing their ability to make more optimal plans.

  * •

Format: It represents how to formulate the plans made by planners. As shown in
Table 1, there are two formatting ways. The first one is natural language. For
example, in [41], the planning content obtained is “The first thing I did was
use OpenCV’s openpose control model to analyze the pose of the boy in the
image….”, where the plan made is to use “OpenCV’s openpose control model”. The
second one is in the form of programs, like “ image_patch = ImagePatch(image)”
as described in [43], which invokes the ImagePatch function to execute the
planning. There are also hybrid forms, such as [9].

  * •

Inspection & Reflection: It is challenging for an LMAs to consistently make
meaningful and task-completing plans in a complex multimodal environment. This
component aims at enhancing robustness and adaptability. Some research methods
[51, 52] store successful experiences in long-term memory, including the
multimodal states, to guide planning. During the planning process, they first
retrieve relevant experiences, aiding planners in thoughtful deliberation to
reduce uncertainty. Additionally, [12] utilizes plans made by humans in
different states while performing the same tasks. When encountering similar
states, planners can refer to these “standard answers" for contemplation,
leading to more rational plans. Moreover, [71] employs more complex planning
methods, like Monte Carlo, to expand the scope of planning search to find the
optimal planning strategy.

  * •

Planning Methods: Existing planning strategies can be categorized into two
types: dynamic and static planning as shown in Table 1. The former [57, 43,
70, 30, 41] refers to decomposing the goal into a series of sub-plans based on
the initial input, similar to Chain of Thought (CoT) [80], where plans are not
reformulated even if errors occur during the process; The latter [9, 25, 51,
71] implies that each plan is formulated based on the current environmental
information or feedback. If errors are detected in the plan, it will revert to
the original state for re-planning [12].

#### Action.

The action component in multimodal agent systems is responsible for executing
the plans and decisions formulated by the planner. It translates these plans
into specific actions, such as the use of tools, physical movements, or
interactions with interfaces, thereby ensuring that the agent can achieve its
goals and interact with the environment accurately and efficiently. Our
discussion focuses on two aspects: types and approaches.

Actions in Table 1 are classified into three categories: tool use (T),
embodied actions (E), and virtual actions (V), where tools includes visual
foundation models (VFMs), APIs, Python, etc (as listed in Table 2); Embodied
actions are performed by physical entities like robots [32, 7] or virtual
characters [51, 52, 45, 68]; Virtual actions [8, 76, 44, 54] include web tasks
(e.g., clicking links, scrolling, and keyboard use). In terms of approaches,
as shown in Table 1, there are primarily two types. The first type involves
using prompts to provide agents with information about executable actions,
such as the tools available at the moment and their functions; The second type
involves collecting data on actions and leveraging this information to self-
instruct the fine-tuning process of open-source large models, such as LLaVA
[23]. This data is typically generated by advanced models, such as GPT-4.
Compared to language-only agents, the complexity of information and data
related to actions requires more sophisticated methods to optimize the
learning strategy.

Modality Skill Tools Source Image VQA BLIP2 [18] Github, HuggingFace
Grounding/Detection G-DINO [24] Github, HuggingFace Image Caption BLIP
[19],BLIP2 [18],InstructBLIP [5] Github, HuggingFace, API OCR EasyOCR,Umi-OCR
Github, API Image Editing Instruct P2P [63] Github, HuggingFace, API Image
Generation Stable Diffusion [39], DALLE·3 [1] Github, HuggingFace, API Image
Segmentation SAM [15], PaddleSeg [28] Github, HuggingFace, API Text Knowledge
Retrieval Bing Search Website, API Programming Related Skill PyLint, PyChecker
Python, API Video Video Editing Editly Github, API Object-tracking OSTrack
[72] Github, HuggingFace, API Audio Speech to Text Whisper [2] Github,
HuggingFace, API Text to Speech StyleTTS 2 [22] Github, API

Table 2: A summary of different tools, including their corresponding
modalities, skills, and available sources.

#### Memory.

Early studies show that memory mechanisms play a vital role in the operation
of general-purpose agents. Similar to humans, memory in agents can be
categorized into long and short memory. In a simple environment, short memory
suffices for an agent to handle tasks at hand. However, in more complex and
realistic settings, long memory becomes essential. In Table 1, we can see that
only a minority of LMAs incorporate long memory. Unlike language-only agents,
these multimodal agents require long memory capable of storing information
across various modalities. In some studies [71, 47, 69, 7], all modalities are
converted into textual formats for storage. However, in [51], a multimodal
long memory system is proposed, designed specifically to archive previous
successful experiences. Specifically, these memories are stored as key-value
pairs, where the key is the multimodal state and the value is the successful
plan. Upon encountering a new multimodal state, the most analogous examples
are retrieved based on their encoded similarity:

$p(t|x)\propto\text{CLIP}_{v}(k_{t})^{\top}\text{CLIP}_{v}(k_{x}),$ (1)

where $k_{t}$ represents the key’s visual information encoded via the CLIP
model, compared for similarity with the current visual state $k_{x}$, also
encoded by CLIP.

##  3 The Taxonomy of LMAs

In this section, we present a taxonomy of existing studies by classifying them
into four types.

#### Type I: Closed-source LLMs as Planners w/o Long-term Memory.

Early studies [11, 43, 57, 41, 9, 25] employ prompts to utilize closed-source
large language models (e.g., GPT-3.5) as the planner for inference and
planning as illustrated in Figure 2(a). Depending on the specific environment
or task requirements, the execution of these plans may be carried out by
downstream toolkits or through direct interaction with the environment using
physical devices like mice or robotic arms. LMAs of this type typically
operate in simpler settings, undertaking conventional tasks such as image
editing, visual grounding, and visual question answering (VQA).

#### Type II: Finetuned LLMs as Planners w/o Long-term Memory.

LMAs of this type involve collecting multimodal instruction-following data or
employing self-instruction to fine-tune open-source large language models
(such as LLaMA) [67] or multimodal models (like LLaVA) [23, 46], as
illustrated in Figure 2(b). This enhancement not only allows the models to
serve as the central “brain" for reasoning and planning but also to execute
these plans. The environments and tasks faced by Type II LMAs are similar to
those in Type I, typically involving traditional visual or multimodal tasks.
Compared to canonical scenarios characterized by relatively simple dynamics,
closed environments, and basic tasks, LMAs in open-world games like Minecraft
are required to execute precise planning in dynamic contexts, handle tasks of
high complexity, and engage in lifelong learning to adapt to new challenges.
Therefore, building upon the foundation of Type I and Type II, Type III and
Type IV LMAs integrate a memory component, showing great promise in developing
towards a generalist agent in the field of artificial intelligence.

#### Type III: Planners with Indirect Long-term Memory.

For Type III LMAs [71, 47], as illustrated in Figure 2(c), LLMs function as
the central planner and are equipped with long memory. These planners access
and retrieve long memories by invoking relevant tools, leveraging these
memories for enhanced reasoning and planning. For example, the multimodal
agent framework developed in [71] is tailored for dynamic tasks such as video
processing. This framework consists of a planner, a toolkit, and a task-
relevant memory bank that catalogues spatial and temporal attributes. The
planner employs specialized sub-task tools to query the memory bank for
spatiotemporal attributes related to the video content, enabling inference on
task-relevant temporal and spatial data. Stored within the toolkit, each tool
is designed for specific types of spatiotemporal reasoning and acts as an
executor within the framework.

#### Type IV: Planners with Native Long-term Memory.

Different from Type III, Type IV LMAs [51, 37, 7, 76] feature LLMs directly
interacting with long memory, bypassing the need for tools to access long
memories, as illustrated in Figure 2(d). For example, the multimodal agent
proposed in [51] demonstrates proficiency in completing over 200 distinct
tasks within the open-world context of Minecraft. In their multimodal agent
design, the interactive planner, merging a multimodal foundation model with an
LLM, first translates environmental multimodal inputs into text. The planner
further employs a self-check mechanism to anticipate and assess each step in
execution, proactively spotting potential flaws and, combined with
environmental feedback and self-explanation, swiftly corrects and refines
plans without extra information. Moreover, this multimodal agent framework
includes a novel multimodal memory. Successful task plans and their initial
multimodal states are stored, and the planner retrieves similar states from
this database for new tasks, using accumulated experiences for faster, more
efficient task completion.

##  4 Multi-agent Collaboration

We further introduce the collaborative framework for LMAs beyond the
discussion within isolated agents in this section.

As shown in Figure 3(a)(b), these frameworks employ multiple LMAs working
collaboratively. The key distinction between the two frameworks lies in the
presence or absence of a memory component, but their underlying principle is
consistent: multiple LMAs have different roles and responsibilities, enabling
them to coordinate actions to collectively achieve a common goal. This
structure alleviates the burden on a single agent, thereby enhancing task
performance [12, 37, 17, 29].

For example, in Table 1, in the multimodal agent framework by [37], a
perceiver agent is introduced to sense the multimodal environment, comprised
of large multimodal models. An agent, designated as Patroller, is responsible
for engaging in multiple interactions with the perceiver agent, conducting
real-time checks and feedback on the perceived environmental data to ensure
the accuracy of current plans and actions. When execution failures are
detected or reevaluation is necessitated, Patroller provides pertinent
information to the planner, prompting a reorganization or update of the action
sequences under the sub-goals. The MemoDroid framework [17] comprises several
key agents that collaboratively work to automate mobile tasks. The Exploration
Agent is responsible for offline analysis of the target application interface,
generating a list of potential sub-tasks based on UI elements, which are then
stored in the application memory. During the online execution phase, the
Selection Agent determines specific sub-tasks to execute from the explored
set, based on user commands and the current screen state. The Deduction Agent
further identifies and completes the underlying action sequences required for
the selected sub-tasks by prompting an LLM. Concurrently, the Recall Agent,
upon encountering tasks similar to those previously learned, can directly
invoke and execute the corresponding sub-tasks and action sequences from
memory.

##  5 Evaluation

The predominant focus of research is on enhancing the capabilities of current
LMAs. However, limited efforts are devoted to developing methodologies for the
assessment and evaluation of these agents. The majority of research continues
to depend on conventional metrics for evaluating performance, clearly
illustrating the challenges inherent in assessing LMAs. This also underscores
the necessity of developing pragmatic assessment criteria and establishing
benchmark datasets in this domain. This section summarizes existing
evaluations of LMAs and offers perspectives on future developments.

###  5.1 Subjective Evaluation

Subjective Evaluation mainly refers to using humans to assess the capabilities
of these LMAs. Our ultimate goal is to create a LMA that can comprehend the
world like humans and autonomously execute a variety of tasks. Therefore, it
is crucial to adopt subjective evaluations of human users on the capabilities
of LMAs. The main evaluation metrics include versatility, user-friendliness,
scalability, and value and safety.

#### Versatility.

Versatility denotes the capacity of an LMA to adeptly utilize diverse tools,
execute both physical and virtual actions, and manage assorted tasks. [30]
propose comparing the scale and types of tools utilized in existing LMAs, as
well as assessing the diversity of their capabilities.

#### User-Friendliness.

User-friendliness involves user satisfaction with the outcomes of tasks
completed by LMAs, including efficiency, accuracy, and the richness of the
results. This type of assessment is relatively subjective. In [64], human
evaluation of the LMA is essential to precisely assess its effectiveness in
interpreting and executing user instructions.

#### Scalability.

Scalability fundamentally evaluates the capability of LMAs to assimilate new
competencies and address emerging challenges. Given the dynamic nature of
human requirements, it is imperative to rigorously assess the adaptability and
lifelong learning potential of LMAs. For example, the evaluation in [23]
focuses on the proficiency of agents in using previously unseen tools to
complete tasks.

#### Value and Safety.

In addition to the metrics previously mentioned, the “Value and Safety” metric
plays a critical role in determining the practical significance and safety of
agents for human users. While many current evaluations overlook this metric,
it is essential to consider the “Value and Safety” of LMAs. Compared to
language agents, LMAs can handle a wider range of task categories, making it
even more important for them to follow ethical and moral principles consistent
with human societal values.

###  5.2 Objective Evaluation

Objective evaluation, distinct from subjective assessment, relies on
quantitative metrics to comprehensively, systematically, and standardizedly
assess the capabilities of LMAs. It is currently the most widely adopted
evaluation method in multimodal agent research.

#### Metrics.

Metrics play a crucial role in objective assessment. In current multimodal
agent research [43, 70, 9, 71, 12, 57, 30], specific task-related metrics are
employed, such as the accuracy of answers generated by the agent in tasks like
visual question answering (VQA) [10, 43]. However, the traditional task
metrics established prior to the emergence of LLMs are not sufficiently
effective in evaluating llm-powered LMAs. As a result, an increasing number of
research efforts are directed towards identifying more appropriate metrics for
assessment. For instance, in VisualWebArena [16], a specialized assessment
metric is designed to evaluate the performance of LMAs in handling visually
guided tasks. This includes measuring the accuracy of the agent’s visual
understanding of webpage content, such as the ability to recognize and utilize
interactable elements marked by Set-of-Marks for operations and achieving
state transitions based on task objectives, as defined by a manually designed
reward function. Besides, it encompasses the accuracy of responses to specific
visual scene questions and the alignment of actions executed based on visual
information.

#### Benchmarks.

Benchmark represents a testing environment that encompasses a suite of
evaluation standards, datasets, and tasks. It is utilized to assess and
compare the performance of different algorithms or systems. Compared to
benchmarks for conventional tasks [30, 12, 57, 23], SmartPlay [58] utilizes a
carefully designed set of games to comprehensively measure the various
abilities of LMAs, establishing detailed evaluation metrics and challenge
levels for each capability. Contrasting with the approach of using games to
evaluate, GAIA [34] has developed a test set comprising 466 questions and
their answers. These questions require AI systems to possess a range of
fundamental abilities, such as reasoning, processing multimodal information,
web navigation, and proficient tool use. Diverging from the current trend of
creating increasingly difficult tasks for humans, it focuses on conceptually
simple yet challenging questions for existing advanced AI systems. These
questions involve real-world scenarios that necessitate the precise execution
of complex operational sequences, with outputs that are easy to verify.
Similarly, VisualWebArena [16] is a benchmark test suite designed to assess
and advance the capabilities of LMAs in processing visual and textual
understanding tasks on real webpages. There are also other benchmarks[31, 61]
that have effectively tested the capabilities of agents.

##  6 Application

LMAs, proficient in processing diverse data modalities, surpass language-only
agents in decision-making and response generation across varied scenarios.
Their adaptability makes them exceptionally useful in real-world, multisensory
environments, as illustrated in Figure 4.

Figure 4: A variety of applications of LMAs.

#### GUI Automation.

In this application, the objective of LMAs is to understand and simulate human
actions within user interfaces, enabling the execution of repetitive tasks,
navigation across multiple applications, and the simplification of complex
workflows. This automation holds the potential to save users’ time and energy,
allowing them to focus on the more critical and creative aspects of their work
[44, 6, 53, 64, 75, 69, 54, 17, 8]. For example, GPT-4V-Act [6], is an
advanced AI that combines GPT-4V’s capabilities with web browsing to improve
human-computer interactions. Its main goal is to make user interfaces more
accessible, simplify workflow automation, and enhance automated UI testing.
This AI is especially beneficial for people with disabilities or limited tech
skills, helping them navigate complex interfaces more easily.

#### Robotics and Embodied AI.

This application [37, 51, 68, 52, 45, 65, 79] focuses on integrating the
perceptual, reasoning, and action capabilities of robots with physical
interactions in their environments. Employing a multimodal agent, robots are
enabled to utilize diverse sensory channels, such as vision, audition, and
touch, to acquire comprehensive environmental data. For example, the MP5
system [37], is a cutting-edge, multimodal entity system used in Minecraft
that utilizes active perception to smartly break down and carry out extensive,
indefinite tasks with large language models.

#### Game Developement.

Game AI [58, 16] endeavors to design and implement these agents to exhibit
intelligence and realism, thereby providing engaging and challenging player
experiences. The successful integration of agent technology in games has led
to the creation of more sophisticated and interactive virtual environments.

#### Autonomous Driving.

Traditional approaches to autonomous vehicles [33] face obstacles in
effectively perceiving and interpreting complex scenarios. Recent progress in
multimodal agent-based technologies, notably driven by LLMs, marks a
substantial advancement in overcoming these challenges and bridging the
perception gap [32, 7, 81, 55]. [32] present GPT-Driver, a pioneering approach
that employs the OpenAI GPT-3.5 model as a reliable motion planner for
autonomous vehicles, with a specific focus on generating safe and comfortable
driving trajectories. Harnessing the inherent reasoning capabilities of LLMs,
their method provides a promising solution to the issue of limited
generalization in novel driving scenarios.

#### Video Understanding.

The video understanding agents [9, 71] are artificial intelligence systems
specifically designed for analyzing and comprehending video content. It
utilizes deep learning techniques to extract essential information from
videos, identifying objects, actions, and scenes to enhance understanding of
the video content.

#### Visual Generation & Editing.

Applications of this kind [4, 70, 47] are designed for the creation and
manipulation of visual content. Using advanced technologies, this tool
effortlessly creates and modifies images, offering users a flexible option for
creative projects. For instance, LLaVA-Interactive [4] is an open-source
multimodal interactive system that amalgamates the capabilities of pre-trained
AI models to facilitate multi-turn dialogues with visual cues and generate
edited images, thereby realizing a cost-effective, flexible, and intuitive AI-
assisted visual content creation experience.

#### Complex Visual Reasoning Tasks.

This area is a key focus in multimodal agent research, mainly emphasizing the
analysis of multimodal content. This prevalence is attributed to the superior
cognitive capabilities of LLMs in comprehending and reasoning through
knowledge-based queries, surpassing the capabilities of previous models [14,
25, 80]. Within these applications, the primary focus is on QA tasks [41, 57,
70, 30]. This entails leveraging visual modalities (images or videos) and
textual modalities (questions or questions with accompanying documents) for
reasoned responses.

#### Audio Editing & Generation.

The LMAs in this application integrate foundational expert models in the audio
domain, making the editing and creation of music efficient[77, 73, 13, 26].

##  7 Conclusion and Future Research

In this survey, we provide a thorough overview of the latest research on
multimodal agents driven by LLMs (LMAs). We start by introducing the core
components of LMAs (i.e., perception, planning, action, and memory) and
classify existing studies into four categories. Subsequently, we compile
existing methodologies for evaluating LMAs and devise a comprehensive
evaluation framework. Finally, we spotlight a range of current and significant
application scenarios within the realm of LMAs. Despite the notable progress,
this field still faces many unresolved challenges, and there is considerable
room for improvement. We finally highlight several promising directions based
on the reviewed progress:

  * •

On frameworks: The future frameworks of LMAs may evolve from two distinct
perspectives. From the viewpoint of a single agent, development could progress
towards the creation of a more unified system. This entails planners directly
interacting with multimodal environments [71], utilizing a comprehensive set
of tools [30], and manipulating memory directly [51]; From the perspective of
multiple agents, advancing the effective coordination among multiple
multimodal agents for the execution of collective tasks emerges as a critical
research trajectory. This encompasses essential aspects such as collaborative
mechanisms, communication protocols, and strategic task distribution.

  * •

On evaluation: Systematic and standard evaluation frameworks are highly
desired for this field. An ideal evaluation framework should encompass a
spectrum of assessment tasks [58, 16], varying from straightforward to
intricate, each bearing significant relevance and utility for humans. It ought
to incorporate lucid and judicious evaluation metrics, meticulously designed
to evaluate the diverse capabilities of an LMA in a comprehensive, yet non-
repetitive manner. Moreover, the dataset used for evaluation should be
meticulously curated to reflect a closer resemblance to real-world scenarios.

  * •

On application: The potential applications of LMAs in the real world are
substantial, offering solutions to problems that were previously challenging
for conventional models, such as web browsing. Furthermore, the intersection
of LMAs with the field of human-computer interaction [54, 44] represents one
of the significant directions for future applications. Their ability to
process and understand information from various modalities enables them to
perform more complex and nuanced tasks, thereby enhancing their utility in
real-world scenarios and improving the interaction between humans and
machines.

## References

  * [1] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2:3, 2023.
  * [2] Nan Cao, Yu-Ru Lin, Xiaohua Sun, David Lazer, Shixia Liu, and Huamin Qu.  Whisper: Tracing the spatiotemporal process of information diffusion in real time.  IEEE transactions on visualization and computer graphics, 18(12):2649–2658, 2012.
  * [3] Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu.  Large language models are visual reasoning coordinators.  arXiv preprint arXiv:2310.15166, 2023.
  * [4] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng Gao, and Chunyuan Li.  Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing.  arXiv preprint arXiv:2311.00571, 2023.
  * [5] W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023.  arXiv preprint arXiv:2305.06500, 2023.
  * [6] ddupont808.  Gpt-4v-act.  https://github.com/ddupont808/GPT-4V-Act, 2023.  Accessed on .
  * [7] Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao.  Drive like a human: Rethinking autonomous driving with large language models.  arXiv preprint arXiv:2307.07162, 2023.
  * [8] Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al.  Assistgui: Task-oriented desktop graphical user interface automation.  arXiv preprint arXiv:2312.13108, 2023.
  * [9] Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou.  Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn.  arXiv preprint arXiv:2306.08640, 2023.
  * [10] Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li.  Clova: A closed-loop visual assistant with tool usage and update.  arXiv preprint arXiv:2312.10908, 2023.
  * [11] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023.
  * [12] Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi.  Avis: Autonomous visual information seeking with large language model agent.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [13] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al.  Audiogpt: Understanding and generating speech, music, sound, and talking head.  arXiv preprint arXiv:2304.12995, 2023.
  * [14] Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata.  Vision-by-language for training-free compositional image retrieval.  arXiv preprint arXiv:2310.09291, 2023.
  * [15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [16] Yujing Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Mingchong Lim, Po-yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried.  Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.  arXiv preprint arXiv:2401.13649, 2024.
  * [17] Sunjae Lee, Junyoung Choi, Jungjae Lee, Hojun Choi, Steven Y Ko, Sangeun Oh, and Insik Shin.  Explore, select, derive, and recall: Augmenting llm with human-like memory for mobile task automation.  arXiv preprint arXiv:2312.03003, 2023.
  * [18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [20] Sen Li, Ruochen Wang, Cho-Jui Hsieh, et al.  Mulan: Multimodal-llm agent for progressive multi-object diffusion.  arXiv preprint arXiv:2402.12741, 2024.
  * [21] Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d’Autume, Phil Blunsom, and Aida Nematzadeh.  A systematic investigation of commonsense knowledge in large language models.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11838–11855, 2022.
  * [22] Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani.  Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [23] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al.  Llava-plus: Learning to use tools for creating multimodal agents.  arXiv preprint arXiv:2311.05437, 2023.
  * [24] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [25] Xiangyan Liu, Rongxue Li, Wei Ji, and Tao Lin.  Towards robust multi-modal reasoning via model selection.  arXiv preprint arXiv:2310.08446, 2023.
  * [26] Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Meng Cui, Qiushi Huang, Jinhua Liang, Yin Cao, Qiuqiang Kong, Mark D Plumbley, et al.  Wavjourney: Compositional audio creation with large language models.  arXiv preprint arXiv:2307.14335, 2023.
  * [27] Yang Liu, Xinshuai Song, Kaixuan Jiang, Weixing Chen, Jingzhou Luo, Guanbin Li, and Liang Lin.  Multimodal embodied interactive agent for cafe scene.  arXiv preprint arXiv:2402.00290, 2024.
  * [28] Yi Liu, Lutao Chu, Guowei Chen, Zewu Wu, Zeyu Chen, Baohua Lai, and Yuying Hao.  Paddleseg: A high-efficient development toolkit for image segmentation.  arXiv preprint arXiv:2101.06175, 2021.
  * [29] Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong.  Discuss before moving: Visual language navigation via multi-expert discussions.  arXiv preprint arXiv:2309.11382, 2023.
  * [30] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.  Chameleon: Plug-and-play compositional reasoning with large language models.  arXiv preprint arXiv:2304.09842, 2023.
  * [31] Xing Han Lù, Zdeněk Kasner, and Siva Reddy.  Weblinx: Real-world website navigation with multi-turn dialogue.  arXiv preprint arXiv:2402.05930, 2024.
  * [32] Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang.  Gpt-driver: Learning to drive with gpt.  arXiv preprint arXiv:2310.01415, 2023.
  * [33] Markus Maurer, J Christian Gerdes, Barbara Lenz, and Hermann Winner.  Autonomous driving: technical, legal and social aspects.  Springer Nature, 2016.
  * [34] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom.  Gaia: a benchmark for general ai assistants.  arXiv preprint arXiv:2311.12983, 2023.
  * [35] Osonde A Osoba, Raffaele Vardavas, Justin Grana, Rushil Zutshi, and Amber Jaycocks.  Policy-focused agent-based modeling using rl behavioral models.  arXiv preprint arXiv:2006.05048, 2020.
  * [36] Jeff Z Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo Lissandrini, et al.  Large language models and knowledge graphs: Opportunities and challenges.  arXiv preprint arXiv:2308.06374, 2023.
  * [37] Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao.  Mp5: A multi-modal open-ended embodied system in minecraft via active perception.  arXiv preprint arXiv:2312.07472, 2023.
  * [38] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  arXiv preprint arXiv:2307.16789, 2023.
  * [39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [40] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  arXiv preprint arXiv:2302.04761, 2023.
  * [41] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  arXiv preprint arXiv:2303.17580, 2023.
  * [42] Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths.  Cognitive architectures for language agents.  arXiv preprint arXiv:2309.02427, 2023.
  * [43] Dídac Surís, Sachit Menon, and Carl Vondrick.  Vipergpt: Visual inference via python execution for reasoning.  arXiv preprint arXiv:2303.08128, 2023.
  * [44] Heyi Tao, Sethuraman TV, Michal Shlapentokh-Rothman, Derek Hoiem, and Heng Ji.  Webwise: Web interface control and sequential exploration with large language models.  arXiv preprint arXiv:2310.16042, 2023.
  * [45] Sai Vemprala, Shuhang Chen, Abhinav Shukla, Dinesh Narayanan, and Ashish Kapoor.  Grid: A platform for general robot intelligence development.  arXiv preprint arXiv:2310.00887, 2023.
  * [46] Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, jindi Guo, Sixun Dong, Xiaohua Xuan, Zhengxin Li, Lin Ma, and Shenghua Gao.  Mllm-tool: A multimodal large language model for tool agent learning.  arXiv preprint arXiv:2401.10727, 2024.
  * [47] Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang.  Chatvideo: A tracklet-centric multimodal and versatile video understanding system.  arXiv preprint arXiv:2304.14407, 2023.
  * [48] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang.  Mobile-agent: Autonomous multi-modal mobile device agent with visual perception.  arXiv preprint arXiv:2401.16158, 2024.
  * [49] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.  A survey on large language model based autonomous agents.  arXiv preprint arXiv:2308.11432, 2023.
  * [50] Xiaoling Wang and Housheng Su.  Completely model-free rl-based consensus of continuous-time multi-agent systems.  Applied Mathematics and Computation, 382:125312, 2020.
  * [51] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al.  Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.  arXiv preprint arXiv:2311.05997, 2023.
  * [52] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang.  Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.  arXiv preprint arXiv:2302.01560, 2023.
  * [53] Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu.  Empowering llm to use smartphone for intelligent task automation.  arXiv preprint arXiv:2308.15272, 2023.
  * [54] Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li.  Droidbot-gpt: Gpt-powered ui automation for android.  arXiv preprint arXiv:2304.07061, 2023.
  * [55] Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al.  On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving.  arXiv preprint arXiv:2311.05332, 2023.
  * [56] Michael Wooldridge and Nicholas R Jennings.  Intelligent agents: Theory and practice.  The knowledge engineering review, 10(2):115–152, 1995.
  * [57] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  arXiv preprint arXiv:2303.04671, 2023.
  * [58] Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li.  Smartplay: A benchmark for llms as intelligent agents.  arXiv preprint arXiv:2310.01557, 2023.
  * [59] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong.  Os-copilot: Towards generalist computer agents with self-improvement.  arXiv preprint arXiv:2402.07456, 2024.
  * [60] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al.  The rise and potential of large language model based agents: A survey.  arXiv preprint arXiv:2309.07864, 2023.
  * [61] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su.  Travelplanner: A benchmark for real-world planning with language agents.  arXiv preprint arXiv:2402.01622, 2024.
  * [62] Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al.  Openagents: An open platform for language agents in the wild.  arXiv preprint arXiv:2310.10634, 2023.
  * [63] Jiale Xu, Xintao Wang, Yan-Pei Cao, Weihao Cheng, Ying Shan, and Shenghua Gao.  Instructp2p: Learning to edit 3d point clouds with text instructions.  arXiv preprint arXiv:2306.07154, 2023.
  * [64] An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al.  Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation.  arXiv preprint arXiv:2311.07562, 2023.
  * [65] Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al.  Octopus: Embodied vision-language programmer from environmental feedback.  arXiv preprint arXiv:2310.08588, 2023.
  * [66] Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, et al.  Supervised knowledge makes large language models better in-context learners.  arXiv preprint arXiv:2312.15918, 2023.
  * [67] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.  Gpt4tools: Teaching large language model to use tools via self-instruction.  arXiv preprint arXiv:2305.18752, 2023.
  * [68] Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi.  Embodied multi-modal agent trained by an llm from a parallel textworld.  arXiv preprint arXiv:2311.16714, 2023.
  * [69] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu.  Appagent: Multimodal agents as smartphone users.  arXiv preprint arXiv:2312.13771, 2023.
  * [70] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  arXiv preprint arXiv:2303.11381, 2023.
  * [71] Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang.  Doraemongpt: Toward understanding dynamic scenes with large language models.  arXiv preprint arXiv:2401.08392, 2024.
  * [72] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen.  Joint feature learning and relation modeling for tracking: A one-stream framework.  In European Conference on Computer Vision, pages 341–357. Springer, 2022.
  * [73] Dingyao Yu, Kaitao Song, Peiling Lu, Tianyu He, Xu Tan, Wei Ye, Shikun Zhang, and Jiang Bian.  Musicagent: An ai agent for music understanding and generation with large language models.  arXiv preprint arXiv:2310.11954, 2023.
  * [74] Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji.  Craft: Customizing llms by creating and retrieving from specialized toolsets.  arXiv preprint arXiv:2309.17428, 2023.
  * [75] Zhuosheng Zhan and Aston Zhang.  You only look at screens: Multimodal chain-of-action agents.  arXiv preprint arXiv:2309.11436, 2023.
  * [76] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim.  Bootstrap your own skills: Learning to solve new tasks with large language model guidance.  arXiv preprint arXiv:2310.10021, 2023.
  * [77] Yixiao Zhang, Akira Maezawa, Gus Xia, Kazuhiko Yamamoto, and Simon Dixon.  Loop copilot: Conducting ai ensembles for music generation and iterative editing.  arXiv preprint arXiv:2310.12404, 2023.
  * [78] Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang.  How do large language models capture the ever-changing world knowledge? a review of recent advances.  arXiv preprint arXiv:2310.07343, 2023.
  * [79] Zhonghan Zhao, Wenhao Chai, Xuan Wang, Li Boyi, Shengyu Hao, Shidong Cao, Tian Ye, Jenq-Neng Hwang, and Gaoang Wang.  See and think: Embodied agent in virtual environment.  arXiv preprint arXiv:2311.15209, 2023.
  * [80] Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang.  Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.  arXiv preprint arXiv:2310.16436, 2023.
  * [81] Xingcheng Zhou, Mingyu Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C Knoll.  Vision language models in autonomous driving and intelligent transportation systems.  arXiv preprint arXiv:2310.14414, 2023.


# Bound states in the continuum induced via local symmetries in complex
structures

Cheng-Zhen Wang  Wave Transport in Complex Systems Lab, Department of Physics,
Wesleyan University, Middletown, CT-06459, USA  Ulrich Kuhl  Université Côte
d’Azur, CNRS, Institut de Physique de Nice (INPHYNI), 06200, Nice, France
Adin Dowling  Wave Transport in Complex Systems Lab, Department of Physics,
Wesleyan University, Middletown, CT-06459, USA  Holger Schanz  University of
Applied Sciences Magdeburg–Stendal, 39114 Magdeburg, Germany  Tsampikos Kottos
<EMAIL_ADDRESS>Wave Transport in Complex Systems Lab, Department of
Physics, Wesleyan University, Middletown, CT-06459, USA

###### Abstract

Bound states in the continuum (BICs) defy conventional wisdom that assumes a
spectral separation between propagating waves, that carry energy away, and
spatially localized waves corresponding to discrete frequencies. They can be
described as resonance states with infinite lifetime, i.e., leaky modes with
zero leakage. The advent of metamaterials and nanophotonics allowed the
creation of BICs in a variety of systems. Mainly, BICs have been realized by
destructive interference between outgoing resonant modes or exploiting
engineered global symmetries that enforce the decoupling of a symmetry-
incompatible bound mode from the surrounding radiation modes. Here, we
introduce theoretically BICs relying on a different mechanism, namely local
symmetries that enforce a field concentration on a part of a complex system
without implying any global symmetry. We experimentally implement such BICs
using microwaves in a compact one-dimensional photonic network and show that
they emerge from the annihilation of two topological singularities, a zero and
a pole, of the measured scattering matrix. Our alternative for achieving BICs
in complex wave systems may be useful for applications like sensing, lasing,
and enhancement of nonlinear interactions that require high-$Q$ modes.

##  I Introduction

Quantum mechanics books, typically, distinguish between two type of states:
bound states whose discrete energy lies below the continuum threshold
(identified by the asymptotic value of the potential at infinity) and
unbounded scattering states with an energy inside the continuum. Examples
where these two categories of states appear include electrons in the presence
of finite potential wells, quantum dots, or atomic potentials. An exception to
the quantum communis intellectus are bound states in the continuum (BICs) Hsu
_et al._ (2016); Joseph  _et al._ (2021); Azzam and Kildishev (2021); Sadreev
(2021). These are spatially bounded solutions of the Schrödinger equation,
with discrete eigenvalues lying inside the continuum of states that propagate
to infinity. They were originally introduced a century ago by von Neumann and
Wigner, using an inverse potential engineering approach von Neumann and Wigner
(1993). The method assumed a square-integrable BIC wavefunction with a
spatially decaying envelope and, using this as a starting point, they tailored
a suitable 3D potential where this wavefunction is an eigenmode. Such “custom-
made” potentials are unrealistic as they are oscillatory in space while
decaying to infinity as a power law and they have never been implemented
experimentally.

BICs are not a quantum phenomenon but rather pertain to all wave systems. This
observation extended the search for BICs to a variety of other platforms
including electromagnetic, acoustic, water or elastic waves (for a review see
Hsu  _et al._ (2016); Sadreev (2021)). Among the various areas, optics and
photonics have undoubtedly been the tip of the spear as far as novel
realizations of BICs are concerned Zhen  _et al._ (2014); Guo  _et al._
(2017); Hsu  _et al._ (2016); Joseph  _et al._ (2021); Azzam and Kildishev
(2021); Marinica  _et al._ (2008); Plotnik  _et al._ (2011); Wang  _et al._
(2022); Doeleman  _et al._ (2018). For example, an inverse design scheme based
on supersymmetric transformations has been implemented in coupled optical
waveguide arrays to engineer the hopping rate between nearest resonators in
order to support BICs Corrielli  _et al._ (2013).

Figure 1:  Experimental setup of a microwave graph detecting bound states in
the continuum (BICs) based on local symmetries.  (a) Schematic representation
of a complex network supporting a BIC due to the presence of a subnetwork
(cycle) with a local symmetry (red-shaded equilateral pentagon). (b) Schematic
representation of the network used in our experiment. A ”tetrahedron” contains
an equilateral triangle where one side length can be varied. The network is
opened by two attached scattering channels. (c) The microwave tetrahedron
network used in our experiments. Coaxial cables are connected by T-junctions
at each of the vertices $n=1,2,\dots,6$. Between vertices $5$ and $6$, we
replace the cable with a phase shifter allowing us to vary the effective
length between these vertices. The reflection amplitudes $r_{11}$, $r_{22}$
and the transmission amplitudes $t_{12}$, $t_{21}$ are measured using a vector
network analyzer (VNA) connected to the vertices 1 and 2, respectively.

Other, more efficient, BIC schemes have been also developed and experimentally
implemented thanks to the advent of metamaterials and nanotechnology Hsu  _et
al._ (2016); Joseph  _et al._ (2021); Azzam and Kildishev (2021). Perhaps the
most prominent approach that also highlights the wave nature of the
phenomenon, is associated with “accidental” BICs Friedrich and Wintgen (1985a,
b). In this case, the parameters of the target system are fine-tuned to
achieve cancellation of outgoing waves to the continuum. A special case of
accidental BIC is associated with Fabry-Perot (FP) configurations Marinica
_et al._ (2008); Hsu  _et al._ (2013); Weimann  _et al._ (2013) of two
identical resonances which interact strongly through the same radiation
channel (e.g. a waveguide). A third mechanism that leads to BICs is invoking
structures with geometric symmetries Robnik (1986); Hsu  _et al._ (2016);
Sadreev (2021). In this case, a trapped mode with a given symmetry can be
embedded into a continuum of states with a distinct symmetry, ensuring the
decoupling of the trapped mode and thus the suppression of its leakage.

Most of the above BIC proposals have been realized in extended structures. In
fact, there is a non-existence theorem for BICs in compact structures Hsu  _et
al._ (2016) (see, however, Refs Ławniczak  _et al._ (2021); Deriy  _et al._
(2022) for exceptions). Finding a genuine BIC in compact systems is a
challenging fundamental problem, and its solution would allow the
implementation of high-$Q$ resonators having a broad range of potential
applications including lasers, sensors, filters and low-loss fibers Hsu  _et
al._ (2016); Sadreev (2021).

Here we provide an alternative approach for the implementation of BICs which
is based on local symmetries (see Fig. 1a). As opposed to the symmetry-
protected BICs, we study structures lacking geometric symmetries. The proposed
mechanism relies on the existence of an embedded symmetric subdomain and
reveals another backdoor in the non-existence theorem for BICs in compact
structures. In contrast to accidental BICs Hsu  _et al._ (2016); Sadreev
(2021), our construction relies on a stringent blueprint and allows to
analytically predict and control the position of BICs in the spectrum.
Moreover, it leads to an infinite ladder of BIC states which occur
periodically in $k-$space. We experimentally demonstrate the new scheme using
a complex network of coaxial microwave cables coupled together via
T-junctions, see Figs. 1b,c. The analysis of the experimental scattering
matrix demonstrates that the formation of these BICs originates from the
coalescence of two topological defects with opposite charges: a zero (with
charge $+1$) and a pole (with charge $-1$) of the scattering matrix.
Experimental implementations of such networks have been reported in a variety
of frameworks including acoustics Röntgen  _et al._ (2023), microwaves Hul
_et al._ (2004); Rehemanjiang  _et al._ (2016); Chen and Anlage (2022); Lu
_et al._ (2020); Allgaier  _et al._ (2014); Chen  _et al._ (2020), photonic
crystal waveguides Ma  _et al._ (2022) and optics Nasari  _et al._ (2023);
Lepri  _et al._ (2017).

##  II Physical mechanism for local-symmetry protection of BICs

Let us explain the local-symmetry protection mechanism using complex photonic
networks as an example (Fig. 1a): the subdomain (subnet) that possesses a
local symmetry is formed by a closed loop (ring) of $N_{l}$ equilateral edges
such that mirror symmetry (with respect to the vertices defining the subnet)
and discrete rotation invariance is guaranteed (see, for example, the violet
pentagon in Fig. 1a with $N_{l}=5$ or the violet triangle in Fig. 1b with
$N_{l}=3$ having a discrete rotational symmetry $C_{5v},C_{3v}$,
respectively). Note that in the case of a network that is formed by photonic
waveguides, the geometric shape of the edges and the angles between them are
typically irrelevant, see Fig. 1c. In this case the discrete rotation must not
be interpreted in physical space. For example, in case of Fig. 1b, an angle is
defined as $2\pi x/3\ell$ where $x$ is a continuous path length along the
cycle and $x=0$ coincides with a vertex. Then the subnet that supports a BIC
is invariant under the transformations $x\to x+\ell$ and $x\to-x$, generating
the symmetry group $C_{3v}$.

Consider the ring first without any connections to the remaining network.
Then, choosing the appropriate symmetry class, there are rotation invariant
eigenfunctions which are antisymmetric with respect to the vertices. That is,
we have eigenfunctions vanishing at all vertices of the subnet. Thus, when the
remaining network is coupled to some or all vertices of the subnet via
coupling constants (not necessarily equal at all vertices), these eigenstates
remain unaltered. In particular, if the remaining network is open and has a
continuous spectrum, the constructed eigenfunctions will be BICs. This
conclusion applies only to the subset of eigenstates pertaining to the
appropriate symmetry class. All other states of the subnet will be strongly
mixed with the states of the remaining network and in the large network limit
the overwhelming majority of states will be ergodically distributed over the
whole network, as expected for typical systems with wave chaos Haake  _et al._
(2019).

Note that the term local symmetry does not imply a subnet with a small total
bond-length; rather it refers to the fact that it involves a subset of
connected edges. In particular, the BIC might be supported by a subnet that
connects distant parts of the total network either by involving a few long
edges or many short ones.

While pure BICs are completely decoupled and do not contribute to transport
across the network, any small perturbation of the subnet will create quasi-
BICs which act as channels across the network and thus strongly affect its
transport properties. The above BIC-mechanism is independent of specific
properties of the network like the precise boundary conditions at the vertices
or their valency (= the number of connected edges). Most importantly it does
not require any geometric symmetry of the network as a whole.

Figure 2:  The transmittance and reflectance spectrum versus the phase shifter
length variation $\delta$ and frequency $f$. (a1) The experimental
measurements of the transmittance through the graph (input from lead $1$,
transmittance is the same as that from lead $2$ due to reciprocity) versus the
input frequency and phase-shifter length variation $\delta$. (a2) and (a3) the
same as in (a1) but for the theoretical modeling with adjusted loss and
without loss, respectively. (b1)-(b3) The reflectance from lead 1 versus the
input frequency and phase-shifter variation $\delta$ for experimental
measurement, theoretical modeling with loss, and theoretical modeling without
loss, respectively. (c1)-(c3) The same plot as in (b1)-(b3) for reflectance
from lead 2. (d)-(f) The cross-section plot of transmittance $T$ for
$\delta=6$, $0$, and $-6$ mm respectively. The blue solid, red dotted and
black solid lines are for the experimental measurement, theoretical modeling
with fitted loss and theoretical modeling without loss, respectively.

##  III Transport in Complex Networks and BIC formation

### Scattering on Complex Networks —

We study the scattering on a complex network of $n=1,\cdots,N$ vertices, where
two vertices $n,m$ may be connected by an edge $E=(n,m)$ with length $l_{E}$.
The position $x_{E}=x$ on edge $E$ is $x=0(l_{E})$ on vertex $n(m)$. The wave
$\psi_{E}(x)$ on the edge $E$ satisfies the Helmholtz equation

$\frac{d^{2}}{dx^{2}}\psi_{E}(x)+k^{2}\psi_{E}(x)=0\,,$ (1)

where $k=\omega n_{r}/c_{0}$ is the wave number, $\omega$ is the angular
frequency, $c_{0}$ is the speed of light, and $n_{r}$ is the complex-valued
relative index of refraction that includes the losses of the coaxial cables.
The solution of Eq. (1) is $\psi_{E}(x)=\phi_{n}\frac{\sin k(l_{E}-x)}{\sin
kl_{E}}+\phi_{m}\frac{\sin kx}{\sin kl_{E}}$, where $\psi_{E}(0)=\phi_{n}$ and
$\psi_{E}(l_{E})=\phi_{m}$ are the values of the field at the vertices. We
turn the compact network to a scattering set-up by attaching transmission
lines (TLs) $\alpha=1,\cdots,L$ to a subset of the vertices. The field on the
$\alpha$th TL takes the form $\psi_{\alpha}(x)={\cal I}_{\alpha}e^{-ikx}+{\cal
O}_{\alpha}e^{+ikx}$ for $x\geq 0$ where $x=0$ is the position of the vertex.
The coefficients ${\cal I}_{\alpha}({\cal O}_{\alpha})$ indicate the amplitude
of the incoming (outgoing) wave on the TLs. At each vertex $n$, continuity of
the wave and current conservation are satisfied. These conditions can be
expressed in a compact form as Kottos and Smilansky (2000)

$(M+iW^{T}W)\Phi=2iW^{T}{\cal I}\,,$ (2)

where $\Phi=(\phi_{1},\phi_{2},\cdots,\phi_{N})^{T}$. The $L$ dimensional
vector ${\cal I}$ contains the amplitudes ${\cal I}_{\alpha}$ of the incident
field, while $W$ is a $L\times N$ matrix describing the connection between the
TLs and the vertices. A matrix element $W_{\alpha,n}$ is 1, if the $\alpha$th
TL is attached to vertex $n$ and zero otherwise. The $N\times N$ matrix $M$

$\displaystyle M_{nm}=\left\\{\begin{array}[]{ll}-\sum_{l\neq n}{\cal
A}_{nl}\cot kl_{nl},&\textrm{if }n=m\\\ {\cal A}_{nm}\csc kl_{nm},&\textrm{if
}n\neq m\end{array}\right.$ (5)

incorporates information about the metric (length of edges) and the
connectivity of the network, where ${\cal A}$ is the adjacency matrix having
elements ${\cal A}_{nm}=1$ whenever two vertices $m,n$ are connected and
${\cal A}_{nm}=0$ otherwise.

The scattering field $\Phi$ on the compact part of the graph can be evaluated
by solving Eq. (2) for $\Phi$. The same expression, together with the
continuity condition at the vertices, where the TLs are attached, can be used
for deriving the scattering matrix Kottos and Smilansky (2000)

$S(k)=-\hat{I}+2iW\frac{1}{M(k)+iW^{T}W}W^{T}\,.$ (6)

For wavenumbers which are integer multiples of $\pi/l_{nl}$ the terms $M_{nm}$
diverge; this can be rectified by appropriate manipulation of the divergent
terms, see Kottos and Smilansky (1999a).

The poles of the scattering matrix in the complex $k$-plane are related to
resonances i.e. purely outgoing solutions of the wave equation and they are
found from the condition $\det(M(k_{p})+iW^{T}W)=0$. Of interest are also the
zeroes of the scattering matrix defined via the secular equation $\det
S(k_{z})=0$. For lossless structures, causality implies $k_{z}=k_{p}^{*}$. The
zeroes $S(k)=0$ correspond to a special type of wavefronts with time-modulated
amplitude, known as coherent virtual absorption, which are temporarily trapped
inside the structure without any leakage Baranov  _et al._ (2017); Longhi
(2018). When Ohmic losses are also included, one can find parameters of the
structure for which the complex zeroes cross the real axis. In this case,
there are stationary, perfectly impedance-matched input wavefronts which are
completely absorbed by the (weakly) lossy elements in the structure which acts
as an interferometric trap, known as coherent perfect absorber Chong  _et al._
(2010). The exceptional case where the poles are equal to the zeroes of the
$S$-matrix corresponds to BIC states which contain neither an incoming nor an
outgoing radiation component and exist at a real frequency of passive
structures. Thus a BIC is invariant under time reversal and it does not affect
the on-shell scattering matrix $S(k)$ since it is decoupled from the far-field
radiation. In the topological-defect picture in the complex $k$-plane a BIC
implies that a charge $+1$ ($S$-matrix zero) annihilates with a charge $-1$
(resonance) on the real axis. We have confirmed experimentally this
topological feature (see below).

Figure 3:  Transmittance spectra at frequencies that are multiples of the
fundamental BIC frequency. (a1)-(a3) The transmittance as a function of input
frequency and bond variation $\delta$ for the experimental measurement (a1),
the theoretical modeling with loss (a2) and without loss (a3) for the 4th BIC
state at $4\times 0.86547$ GHz $\approx 3.46$ GHz). (b1)-(b3) The same as in
(a1)-(a3) but for the 5th BIC state at $4.33$ GHz.

### BIC and quasi BIC states –

We now identify the conditions that are required for the realization of
locally symmetric BIC states on a complex network. To this end, we consider a
subgraph consisting of $C$ edges that form a closed loop within the network
(for example, in Fig. 1a, $C=5$ for the violet pentagon). We assume that they
have equal lengths $\ell$ and construct a state
$\psi_{c}(x)=\sqrt{(2/C\ell)}\sin(k_{M}x)$ which is restricted to this cycle.
This requires $k_{M}=M\pi/\ell$ and $k_{M}=2M\pi/\ell$ ($M=1,2\dots$) for even
and odd $C$, respectively. The above wave function is zero at the vertices
along the subgraph. Therefore the state is not affected by further edges
attached to these vertices which couple the subgraph to the surrounding
network and to the continuum. This argument is completely independent of the
topology of the rest of the network, the number of extra edges that are
attached to the vertices of the cycle, and the number of attached TLs. Further
considerations reveal that it is possible to relax the assumption of equal
edge lengths $\ell$ on the subgraph to rationally related lengths.
Importantly, the above construction is just a special case of the symmetry
argument given in the introduction which may apply also in different
situations.

In an experiment, the BICs are manifested as long-lived quasi-bound (resonant)
states that disappear and reappear in a characteristic manner when the edge
lengths along the subgraph are changed by a small amount $\delta$. The values
of the wave field at the connections to the remaining network and the TLs are
now $\psi_{c}(0)\sim k\delta$. Due to this coupling, a BIC gives rise to a
Feshbach resonance with width $\gamma\sim|\delta|^{2}$. The latter reveals
itself as a sharp feature in the transmittance $T(f)$ and left (right)
reflectance $R_{1}(f)$ ($R_{2}(f)$), where $R_{1}=R_{2}$ if there are no
losses inside the network. This feature is absent when $\delta=0$ and this is
an indirect, but experimentally observable, signature of a BIC in scattering
data.

##  IV Experimental Implementation

We consider the tetrahedron network shown in Figs. 1b,c. This network is
relatively simple and it does not have any geometrical symmetries. At the same
time, the dynamics is rich enough to show typical features of wave chaos Haake
_et al._ (2019); Kottos and Smilansky (1999b, 2000); Schanz and Kottos (2003).
Its microwave implementation is done using coaxial cables (Huber+Suhner S
04272) connected by 6 T-junctions (vertices). The electrical permittivity of
the cables was found to be $\epsilon\approx 1.56(\pm 0.07)+i0.0015(\pm
0.0005)$ indicating the presence of uniform losses. Two TLs have been attached
at the vertices labeled $1$ and $2$, see Figs. 1b,c.

We choose the triangle consisting of the vertices 4, 5, and 6 as the closed
loop that supports BICs. The electrical lengths $l_{45}=l_{46}=\ell=346.6$ mm
are fixed while the length $l_{56}$ of the third edge varies such that
$l_{56}=\ell+\delta$ with $\delta\in[-14,30]$ mm (see Supplementary Material).
The length-variation is done using a phase shifter whose effective length is
controlled electronically. Following the theoretical arguments of the previous
section we expect that BICs will appear at $\delta=0$, for wavenumbers
$k_{M}=2M\pi/\ell$, $M\in\mathbb{N}^{+}$ corresponding to frequencies
$f_{M}=M\frac{c_{0}}{\ell}=M\times 0.86547$ GHz. The other lengths of the
network are $l_{13}=l_{25}=18.2$ mm, $l_{23}=424.2$ mm, $l_{16}=926.5$ mm, and
$l_{34}=831.4$ mm.

In Figs. 2a1, b1, and c1, we report the measured transmittance $T$ and
reflectances $R_{1}$ and $R_{2}$ versus the input frequency and the tuning
parameter $\delta$ in the proximity of the 2nd BIC frequency ($2\times
0.86547$ GHz $\approx 1.73$ GHz) indicated by the arrow. The second (third)
column reports our calculations for the network of Fig. 1b,c in the presence
(absence) of Ohmic losses at the coaxial cables. In the frequency range
$f\in[1.6,1.8]$ GHz, a BIC is predicted at $f_{2}\approx 1.73$ GHz. In all
cases we find a resonance moving linearly from around 1.7 GHz at $\delta=15$
mm to 1.76 GHz at $\delta=-10$ mm. At $\delta=0$ mm a BIC is formed and the
resonance feature disappears from all three spectra ($T(f)$ and $R_{1,2}(f)$).
This is the expected indirect signature of the BIC: since the state is
completely decoupled from the rest of the network and the TLs, any incident
radiation cannot excite it and therefore there are no signatures of its
existence in the scattering matrix elements. Instead, for small $\delta\neq
0$, the transmittance $T(f\sim f_{M})$ and reflectance $R(f\sim f_{M})$ show a
narrow resonance structure in their frequency-dependence whose width is
controlled by the parameter $\delta$. All these features of the transmittance
and reflectance spectra are present in our measurements and in both sets of
calculations (with and without losses). The presence of losses, however,
smooths out some of the sharp characteristics of the (quasi-)BIC resonance
(first and second columns) which are much more pronounced in the calculations
shown in the third column where we have considered the same network without
any losses of the coaxial cables.

To further investigate the variations of the resonance features in $T(f)$ as
the tuning parameter changes around $\delta=0$ we plot in the right column of
Fig. 2, the transmittance spectrum for three different $\delta$ values around
the BIC value. For $\delta=6$ mm (Fig. 2d) a narrow resonance dip (indicated
by the red arrow) is evident in both, measurements (blue solid line) and
calculations (red dotted line), where the Ohmic losses of the cables are taken
into account. This dip becomes very sharp in the case of a lossless network
modeling (black solid line). When $\delta=0$ mm, (Fig. 2e), the resonance dip
disappears in all cases and reappears again when $\delta=-6$ mm, (Fig. 2f),
after acquiring a small blue-shift.

We have also confirmed experimentally the appearance of a cascade of bound
states at predetermined $k$-values which are multiples of a “fundamental” BIC
frequency and distinguish our approach from accidental BICs. To demonstrate
this feature, we report in Fig. 3 the transmittance versus frequency and
$\delta$ in different frequency regions where we expect the BIC to occur,
i.e., $f=3.35-3.55$ GHz, and $f=4.24-4.44$ GHz. The same behavior as the one
found for the BIC presented in Fig. 2 is observed here as well. Namely, we
observe the appearance and disappearance of a quasi-BIC mode as $\delta$
varies, signifying the formation of BICs at $\delta=0$.

##  V Poles, zeros, and the topological structure of BICs

Figure 4:  Formation of a BIC with local symmetry via the annihilation of a
zero and a pole pair. (a) Parametric evolution of poles (red) and zeros (blue)
of the scattering matrix over the cable length $l_{56}$ of the tetrahedron
network of Fig. 1b,c for $\delta\in[-12,12]$ mm. The experimental poles (red
crosses) zeros (blue-filled circles) have been extracted from the measured
$S$-matrix. The numerical results are indicated with lines of the respective
color and have been extracted from the $S$-matrix given by Eq. 6. All data
have been shifted along the ${\cal I}m(k)$ axis by $0.00314$ due to the global
loss induced by the Ohmic resistances in the coaxial cables. The
$f_{scar}=1.7309$ GHz corresponding to the BIC in Fig. 2. (b) Streamlines of
the vector field $\vec{\Psi}=\nabla_{\vec{k}}arg\left(\det(S(\vec{k})\right)$
with $\vec{k}=(\textrm{Re}(k),\textrm{Im}(k))$, showing $+1$ and $-1$
topological charges at zeros and poles of the $S$-matrix corresponding to a
$\delta=-6$ mm.

Finally, we analyze the formation of BICs from a topological perspective
Sweeney  _et al._ (2020) by analyzing the parametric evolution of the poles
and zeros in the complex frequency plane as $\delta$ is changing. In Fig. 4a
the symbols represent the poles (red crosses) and zeros (blue-filled circles)
of the $S$-matrix that have been extracted from the measured $S$-matrix using
the harmonic inversion method Kuhl  _et al._ . The method was applied to the
off-diagonal elements of the $S$-matrix for the poles and to the matrix
$S^{-1}$ for the zeroes. Our data indicate their coalescence at $\delta=0$ and
at the BIC-frequency $f_{BIC}$ as discussed above. The solid lines correspond
to the simulations. The formation of the BIC in the complex
${\vec{k}}=\large(\textrm{Re}(k);\textrm{Im}(k)\large)$ plane, is a
consequence of the annihilation of two topological defects of the secular
function $\det S({\vec{k}})$ Sweeney  _et al._ (2020): a zero ${\vec{k}}_{z}$
and a pole ${\vec{k}}_{p}$ of the scattering matrix which collides as the
perturbation parameter $\delta$ vanishes (see Fig. 4a). The zero (pole) is
characterized by the topological charge $q\equiv\frac{1}{2\pi}\oint
d{\vec{k}}\cdot\nabla_{k}\phi({\vec{k}})=+1$ ($q=-1$) which describes how many
times the total phase of the scattering matrix $\phi(k)=\arg(\det S(k))$ winds
by $2\pi$ along a counterclockwise simple closed path that encloses the
topological defect (see Fig. 4b). A non-zero charge $q\neq 0$ indicates that a
zero or a pole cannot suddenly disappear when the parameter $\delta$ slightly
varies, although they can move in the complex ${\vec{k}}$-plane. Only a
collision of a $+1$ charge with a $-1$ charge can result in their mutual
annihilation, which signifies that at this parameter value $\delta$ a
resonance mode and a zero mode coexist.

##  VI Conclusion

We have revealed a physical mechanism based on local symmetries that leads to
the creation of BICs in compact photonic networks without any geometrical
symmetry. The resulting BICs are formed as a consequence of the collision of
two topological defects (a pole and zero of the scattering matrix) which leads
to their annihilation. The proposed BICs differ from scenarios, where BICs are
formed due to the presence of a global symmetry or where they are the
”accidental” result of a parameter variation. In particular, they do not
require the degeneracy of two (or more) resonant modes and they are based on a
precise rule that allows to construct and control a ladder of BIC states at
multiples of a fundamental frequency. Our microwave demonstration can be
extended to other wave platforms ranging from optics to acoustics and water
waves and provides a new way of achieving and utilizing BICs in complex
systems.

## References

  * Hsu  _et al._ (2016) C. W. Hsu, B. Zhen, A. D. Stone, J. D. Joannopoulos, and M. Soljacić, “Bound states in the continuum,” Nature Reviews Materials 1, 1 (2016).
  * Joseph  _et al._ (2021) Shereena Joseph, Saurabh Pandey, Swagato Sarkar, and Joby Joseph, “Bound states in the continuum in resonant nanostructures: an overview of engineered materials for tailored applications,” Nanophotonics 10, 4175–4207 (2021).
  * Azzam and Kildishev (2021) Shaimaa I. Azzam and Alexander V. Kildishev, “Photonic bound states in the continuum: From basics to applications,” Advanced Optical Materials 9, 2001469 (2021).
  * Sadreev (2021) Almas F Sadreev, “Interference traps waves in an open system: bound states in the continuum,” Reports on Progress in Physics 84, 055901 (2021).
  * von Neumann and Wigner (1993) J. von Neumann and E. P. Wigner, “Über merkwürdige diskrete eigenwerte,” in  _The Collected Works of Eugene Paul Wigner: Part A: The Scientific Papers_, edited by Arthur S. Wightman (Springer Berlin Heidelberg, Berlin, Heidelberg, 1993) pp. 291–293.
  * Zhen  _et al._ (2014) Bo Zhen, Chia Wei Hsu, Ling Lu, A. Douglas Stone, and Marin Soljačić, “Topological nature of optical bound states in the continuum,” Phys. Rev. Lett. 113, 257401 (2014).
  * Guo  _et al._ (2017) Yu Guo, Meng Xiao, and Shanhui Fan, “Topologically protected complete polarization conversion,” Phys. Rev. Lett. 119, 167401 (2017).
  * Marinica  _et al._ (2008) D. C. Marinica, A. G. Borisov, and S. V. Shabanov, “Bound states in the continuum in photonics,” Phys. Rev. Lett. 100, 183902 (2008).
  * Plotnik  _et al._ (2011) Yonatan Plotnik, Or Peleg, Felix Dreisow, Matthias Heinrich, Stefan Nolte, Alexander Szameit, and Mordechai Segev, “Experimental observation of optical bound states in the continuum,” Phys. Rev. Lett. 107, 183901 (2011).
  * Wang  _et al._ (2022) Jiajun Wang, Lei Shi, and Jian Zi, “Spin hall effect of light via momentum-space topological vortices around bound states in the continuum,” Phys. Rev. Lett. 129, 236101 (2022).
  * Doeleman  _et al._ (2018) Hugo M. Doeleman, Francesco Monticone, Wouter den Hollander, Andrea Alú, and A. Femius Koenderink, “Experimental observation of a polarization vortex at an optical bound state in the continuum,” Nature Photonics 12, 397 (2018).
  * Corrielli  _et al._ (2013) G. Corrielli, G. Della Valle, A. Crespi, R. Osellame, and S. Longhi, “Observation of surface states with algebraic localization,” Phys. Rev. Lett. 111, 220403 (2013).
  * Friedrich and Wintgen (1985a) H. Friedrich and D. Wintgen, “Physical realization of bound states in the continuum,” Phys. Rev. A 31, 3964–3966 (1985a).
  * Friedrich and Wintgen (1985b) H. Friedrich and D. Wintgen, “Interfering resonances and bound states in the continuum,” Phys. Rev. A 32, 3231–3242 (1985b).
  * Hsu  _et al._ (2013) Chia Wei Hsu, Bo Zhen, Song-Liang Chua, Steven G Johnson, John D Joannopoulos, and Marin Soljacić, “Bloch surface eigenstates within the radiation continuum,” Light: Science & Applications 2, e84 (2013).
  * Weimann  _et al._ (2013) Steffen Weimann, Yi Xu, Robert Keil, Andrey E. Miroshnichenko, Andreas Tünnermann, Stefan Nolte, Andrey A. Sukhorukov, Alexander Szameit, and Yuri S. Kivshar, “Compact surface fano states embedded in the continuum of waveguide arrays,” Phys. Rev. Lett. 111, 240403 (2013).
  * Robnik (1986) M. Robnik, “A simple separable hamiltonian having bound states in the continuum,” Journal of Physics A: Mathematical and General 19, 3845 (1986).
  * Ławniczak  _et al._ (2021) Michał Ławniczak, Jiří Lipovskỳ, Małgorzata Białous, and Leszek Sirko, “Application of topological resonances in experimental investigation of a fermi golden rule in microwave networks,” Physical Review E 103, 032208 (2021).
  * Deriy  _et al._ (2022) Ilya Deriy, Ivan Toftul, Mihail Petrov, and Andrey Bogdanov, “Bound states in the continuum in compact acoustic resonators,” Phys. Rev. Lett. 128, 084301 (2022).
  * Röntgen  _et al._ (2023) M Röntgen, O Richoux, G Theocharis, C V Morfonios, P Schmelcher, P del Hougne, and V Achilleos, “Equireflectionality and customized unbalanced coherent perfect absorption in asymmetric waveguide networks,” (2023), arXiv:2305.02786v1 .
  * Hul  _et al._ (2004) O. Hul, S. Bauch, P. Pakoński, N. Savytskyy, K. Życzkowski, and L. Sirko, “Experimental simulation of quantum graphs by microwave networks,” Phys. Rev. E 69, 056205 (2004).
  * Rehemanjiang  _et al._ (2016) A. Rehemanjiang, M. Allgaier, C. H. Joyner, S. Müller, M. Sieber, U. Kuhl, and H.-J. Stöckmann, “Microwave realization of the gaussian symplectic ensemble,” Phys. Rev. Lett. 117, 064101 (2016).
  * Chen and Anlage (2022) Lei Chen and Steven M. Anlage, “Use of transmission and reflection complex time delays to reveal scattering matrix poles and zeros: Example of the ring graph,” Phys. Rev. E 105, 054210 (2022).
  * Lu  _et al._ (2020) Junjie Lu, Jiongning Che, Xiaodong Zhang, and Barbara Dietz, “Experimental and numerical investigation of parametric spectral properties of quantum graphs with unitary or symplectic symmetry,” Phys. Rev. E 102, 022309 (2020).
  * Allgaier  _et al._ (2014) Markus Allgaier, Stefan Gehler, Sonja Barkhofen, H.-J. Stöckmann, and Ulrich Kuhl, “Spectral properties of microwave graphs with local absorption,” Phys. Rev. E 89, 022925 (2014).
  * Chen  _et al._ (2020) L. Chen, T. Kottos, and S.M. Anlage, “Perfect absorption in complex scattering systems with or without hidden symmetries,” Nature Communications 11, 5826 (2020).
  * Ma  _et al._ (2022) Shukai Ma, Thomas M. Antonsen, and Steven M. Anlage, “Eigenfunction and eigenmode-spacing statistics in chaotic photonic crystal graphs,” Phys. Rev. E 106, 054215 (2022).
  * Nasari  _et al._ (2023) Hadiseh Nasari, Georgios G. Pyrialakos, Demetrios N. Christodoulides, and Mercedeh Khajavikhan, “Non-hermitian topological photonics,” Opt. Mater. Express 13, 870–885 (2023).
  * Lepri  _et al._ (2017) Stefano Lepri, Cosimo Trono, and Giovanni Giacomelli, “Complex active optical networks as a new laser concept,” Phys. Rev. Lett. 118, 123901 (2017).
  * Haake  _et al._ (2019) F. Haake, S. Gnutzmann, and M. Kuś,  _Quantum Signatures of Chaos_ (Springer Series in Synergetics, 2019).
  * Kottos and Smilansky (2000) Tsampikos Kottos and Uzy Smilansky, “Chaotic scattering on graphs,” Phys. Rev. Lett. 85, 968 (2000).
  * Kottos and Smilansky (1999a) Tsampikos Kottos and Uzy Smilansky, “Periodic orbit theory and spectral statistics for quantum graphs,” Annals of Physics 274, 76–124 (1999a).
  * Baranov  _et al._ (2017) Denis G. Baranov, Alex Krasnok, and Andrea Alù, “Coherent virtual absorption based on complex zero excitation for ideal light capturing,” Optica 4, 1457–1461 (2017).
  * Longhi (2018) S. Longhi, “Coherent virtual absorption for discretized light,” Opt. Lett. 43, 2122–2125 (2018).
  * Chong  _et al._ (2010) Y. D. Chong, Li Ge, Hui Cao, and A. D. Stone, “Coherent perfect absorbers: Time-reversed lasers,” Phys. Rev. Lett. 105, 053901 (2010).
  * Kottos and Smilansky (1999b) Tsampikos Kottos and Uzy Smilansky, “Periodic orbit theory and spectral statistics for quantum graphs,” Ann. Phys. 274, 76–124 (1999b).
  * Schanz and Kottos (2003) H. Schanz and T. Kottos, “Scars on quantum networks ignore the lyapunov exponent,” Phys. Rev. Lett. 90, 234101 (2003).
  * Sweeney  _et al._ (2020) William R. Sweeney, Chia Wei Hsu, and A. Douglas Stone, “Theory of reflectionless scattering modes,” Phys. Rev. A 102, 063511 (2020).
  * (39) U. Kuhl, R. Höhmann, J. Main, and H.-J. Stöckmann, “Resonance widths in open microwave cavities studied by harmonic inversion,” Phys. Rev. Lett. 100, 254101.

Supplemental Material
for manuscript: “Bound states in the continuum induced via local symmetries in
complex structures”

Cheng-Zhen Wang1, Ulrich Kuhl2, Adin Dowling1, Holger Schanz3, and Tsampikos
Kottos1

1 Wave Transport in Complex Systems Lab, Department of Physics, Wesleyan
University, Middletown, CT-06459, USA

2 Université Côte d’Azur, CNRS, Institut de Physique de Nice (INPHYNI), 06200
Nice, France

3 University of Applied Sciences Magdeburg–Stendal, 39114 Magdeburg, Germany

##  Suppl. material A Experimental Characterization of the Complex Network

To determine the parameters entering the numerical simulation, we have
measured the different cable lengths as $l_{16}=726$ mm, $l_{34}=644$ mm,
$l_{23}=310$ mm, and $l_{45}=l_{46}=245.7$ mm. Have in mind that the real
length is also including lengths from the T junctions which are 18 mm.
Additionally we assume that the phase-shifters optical length is equivalent to
the optical length of cable $l_{45}$ when we have $\delta=0$ mm corresponding
to a fixed real length of the phase shifter. We extract the complex refractive
index $n$ of the cables and phase-shifter based on the transmission
measurements. Comparing the wave propagation through the coaxial cable, to the
Helmholtz equation for a 1D free and uniform medium, we can get the
transmission coefficient $t$ as

$t=\frac{\psi(l)}{A}=e^{i\frac{\omega}{c}\textrm{Re}(n)l}e^{-\frac{\omega}{c}\textrm{Im}(n)l}\,,$
(SA.1)

where we have the refractive index as $n=\sqrt{\epsilon_{0}}$, $\omega=2\pi f$
the angular frequency, and the cable length $l$. First, we can get the real
part of $t$ as

$\textrm{Re}(t)=\cos[\frac{\omega}{c}\textrm{Re}(n)l]e^{-\frac{\omega}{c}\textrm{Im}(n)l}\,,$
(SA.2)

where we can see the $\cos$ terms is oscillation part, which could give us the
$\textrm{Re}(n)$ based on our measured $t$ in a frequency range and measured
cable length $l$. Second, the imaginary part of refractive index $n$ can be
obtained based on the transmittance $T=|t|^{2}$ which is of the form of

$T=|t|^{2}=e^{-\frac{2\omega}{c}\textrm{Im}(n)l}\,,$ (SA.3)

where we can get $\textrm{Im}(n)$ by fitting the curve of transmittance versus
frequency for certain cable length $l$. To avoid the influence of the SMA
connector effect on the cable length, in our measurement, we use the ratio of
transmission $t_{1}/t_{2}$ for two cables with different length $l_{1}$,
$l_{2}$. In this way, we only need the cable length difference so that we can
get rid of the error from the SMA connector and get more accurate
$\textrm{Re}(n)$ and $\textrm{Im}(n)$. By fitting the transmission curves, we
get $\textrm{Re}(n)=1.212$ and $\textrm{Im}(n)=0.002$ for coaxial cables. For
the phase shifter, we obtain the refractive index as $\textrm{Re}(n)=1.004$
and $\textrm{Im}(n)=0.002$. Note, that the $\textrm{Im}(n)$ is frequency
dependent, which we ignored thus taking only an average $n$ for the whole
investigated frequency range into account.

The electrical length of the cables and phase shifter are defined as the
measured geometrical length multiply the real part of refractive index. We
have the $\delta$ to characterize the phase shifter geometrical length shift
with $\delta=0$ corresponding to a total electrical length of phase shifter
equals to be that of $l_{46}$ and increase $\delta$ corresponding increasing
the electrical length of phase shifter.

In our modeling, to get better fit for the theory with experiment, we adopt
the measured cable lengths and the complex refractive index, as well as the T
junction length as the central value, and give each variable a fluctuation
range. Then, we use surrogate optimization method in Matlab to find the
parameter set that gives the best fit for the transmittance and reflectance
curves over the whole $0.5-6.5$ GHz range. The fitted geometrical lengths are
$l_{16}=764.4$ mm, $l_{34}=686$ mm, $l_{23}=350$ mm, $l_{45}=l_{46}=286$ mm,
$l_{13}=l_{25}=15$ mm, refractive index for cable is $n_{c}=1.212+0.0022i$,
and refractive index for phase shifter is $n_{p}=1.004+0.002i$. The slight
difference with respect to the geometrically measured physical lengths of the
cables is attributed to the presence of T-junctions that add an additional
length that has to be included in the modeling and the precision that the
permitivities of the cables have been extracted. For the phase shifter the
electrical length is $\tilde{l}_{56}=286\cdot n_{c}+\delta\cdot n_{p}$, with
$\delta$ the detuning parameter and the imaginary part of the length
representing loss.