# GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging
Cross-Modal Attention with Large Language Models

Haicheng Liao
University of Macau
Macau SAR, China
<EMAIL_ADDRESS>
Both authors contributed equally to this research; †Corresponding author.
Huanming Shen∗
UESTC
Chengdu, China
<EMAIL_ADDRESS>
Zhenning Li†
University of Macau
Macau SAR, China
<EMAIL_ADDRESS>Chengyue Wang
University of Macau
Macau SAR, China
<EMAIL_ADDRESS>Guofa Li
Chongqing University
Chongqing, China
<EMAIL_ADDRESS>Yiming Bie
Jilin University
Jilin, China
<EMAIL_ADDRESS>Chengzhong Xu†
University of Macau
Macau SAR, China
<EMAIL_ADDRESS>
###### Abstract

In the field of autonomous vehicles (AVs), accurately discerning commander
intent and executing linguistic commands within a visual context presents a
significant challenge. This paper introduces a sophisticated encoder-decoder
framework, developed to address visual grounding in AVs. Our Context-Aware
Visual Grounding (CAVG) model is an advanced system that integrates five core
encoders—Text, Image, Context, and Cross-Modal—with a Multimodal decoder. This
integration enables the CAVG model to adeptly capture contextual semantics and
to learn human emotional features, augmented by state-of-the-art Large
Language Models (LLMs) including GPT-4. The architecture of CAVG is reinforced
by the implementation of multi-head cross-modal attention mechanisms and a
Region-Specific Dynamic (RSD) layer for attention modulation. This
architectural design enables the model to efficiently process and interpret a
range of cross-modal inputs, yielding a comprehensive understanding of the
correlation between verbal commands and corresponding visual scenes. Empirical
evaluations on the Talk2Car dataset, a real-world benchmark, demonstrate that
CAVG establishes new standards in prediction accuracy and operational
efficiency. Notably, the model exhibits exceptional performance even with
limited training data, ranging from 50% to 75% of the full dataset. This
feature highlights its effectiveness and potential for deployment in practical
AV applications. Moreover, CAVG has shown remarkable robustness and
adaptability in challenging scenarios, including long-text command
interpretation, low-light conditions, ambiguous command contexts, inclement
weather conditions, and densely populated urban environments. The code for the
proposed model is available at our Github.

_K_ eywords Autonomous Driving $\cdot$ Visual Grounding $\cdot$ Cross-Modal
Attention $\cdot$ Large Language Models $\cdot$ Human-Machine Interaction

##  1 Introduction

While autonomous vehicles (AVs) herald a transformative era in urban mobility
and have the potential to significantly reduce road accidents, their
widespread acceptance remains hampered by public skepticism [1]. The root of
this hesitancy often lies in concerns about trust and the perceived loss of
control. These issues are exacerbated when AVs display indecisiveness or
ambiguity in complex driving scenarios [2]. To address this trust gap, our
research introduces a pioneering framework that augments human-AV interaction
by effectively grounding natural language commands within visual contexts.
Envision a scenario where a commander instructs an AV with a simple command
like "Park in a shaded area on a sunny day." Such interactions not only
empower the commander but also infuse the vehicle’s decision-making process
with essential context, especially in ambiguous or complex driving situations.
Recent studies further highlight the significant enhancement in commander
experience and acceptance when natural language commands facilitate clear
communication of intentions to AVs [3, 4, 5, 6].

However, decoding commander intent poses a substantial challenge for AVs,
primarily due to the intricacies of interactions between commanders and AVs.
To effectively comprehend and act upon these commands, AVs must adeptly
capture the complex semantic relationship between natural language and the
dynamic visual environment. This task is further complicated by the
multifaceted nature of real-world traffic scenes, characterized by ever-
changing visual elements such as traffic agents, lighting conditions, road
markings, and fluctuating weather. These elements, coupled with the inherent
ambiguity of natural language commands, can lead to potentially hazardous
misunderstandings between commanders and AVs, resulting in unsafe driving
maneuvers.

Figure 1: Illustration of Regions Identified by an AV based on a Raw Image and
a Natural Language Command. The blue bounding box represents the ground truth.
The red and yellow bounding boxes correspond to the prediction results from
CAVG with emotion categorization and without emotion categorization,
respectively.

Addressing these challenges necessitates the development of robust models
capable of interpreting and grounding natural language commands amidst this
complex and continuously evolving visual landscape. Moreover, considering the
emotional and psychological dimensions of human-AV interactions is critical
[7]. Natural language commands inherently carry a spectrum of emotional
overtones—from informative and commanding to urgent. Neglecting the emotional
aspect and its fluctuating responses in different traffic contexts could
hinder the ability to discern subtle variations in commands. These emotional
variables, intertwined with language use, significantly influence the
underlying intent and nuances conveyed in commands. Consequently, adopting a
holistic approach that encompasses the emotional dimension of human-AV
interaction is imperative.

Despite considerable progress in this field, with various methodologies and
frameworks proposed [8, 9, 10], existing solutions typically rely on a
segmented process of object identification followed by linguistic alignment.
These methods focus primarily on the bounding boxes of detected objects, often
overlooking essential real-world driving contexts. Furthermore, traditional
approaches usually concentrate on surface-level features, neglecting
underlying elements that could enrich the interpretation and execution of
commands. In contrast, recent advancements [11, 12, 13] have adopted
integrated frameworks that merge textual and visual elements early in the
process but face limitations in handling complex multi-object scenarios. Even
with the introduction of Transformer-based models [14, 15, 16], challenges
remain in navigating intricate real-world situations. Importantly, most state-
of-the-art models [17, 18, 19] in language grounding tend to overlook the
emotional dynamics inherent in human-AV interactions, a crucial aspect that
our research aims to address.

Our model introduces a novel approach that fuses advancements in visual-
language tasks and image-text retrieval. Adopting a sophisticated, two-pronged
strategy, it capitalizes on the strengths of both single-stream and dual-
stream architectures. The first strategy enhances contextual understanding by
integrating contextual cues into the CAVG model, elevating its ability to
interpret and respond to natural language commands in real-world driving
scenarios. The second strategy involves implementing a multimodal prediction
framework, allowing the system to anticipate various plausible outcomes, each
paired with a calculated likelihood. This approach is particularly beneficial
in mitigating risks and uncertainties in complex and ambiguous driving
situations. Additionally, CAVG categorizes subtle emotions inherent in human
language, shaping its responses to align with the psychological and emotional
context presented by the commanders. As illustrated in Fig. 1, CAVG excels at
grounding semantically matched objects in dynamic visual traffic scenes in
response to natural language commands, adapting its responses to the emotional
context of the commander while maintaining contextual accuracy.

In conclusion, our research introduces a novel model that revolutionizes the
way commanders interact with AVs through natural language commands. This
innovative approach promises to transform the landscape of human-AV
interaction significantly, enhancing commander comfort, enriching user
experience, and elevating satisfaction levels. Our model’s distinct
contributions are as follows:

  * •

We have developed a unique approach utilizing a pre-trained model to blend
semantic information with global image features. This results in a cross-modal
vector, which forms the basis of our novel multi-head cross-model attention
mechanism. This mechanism adeptly integrates global context and textual
information, enriching the understanding of each sub-region in the visual
field.

  * •

Our model incorporates an innovative emotion analysis component using
Generative Pretrained Transformer 4 (GPT-4). This allows us to capture the
nuanced emotional content in natural language commands. This feature makes the
interaction between humans and AVs more intuitive and responsive, enabling the
vehicle to understand and react to the commander’s intent in a more human-
centric manner.

  * •

The Cross-Modal Encoder in our model facilitates the extraction of interactive
features between different input types, such as images and commands. Utilizing
a Region-Specific Dynamic (RSD) layer attention mechanism, our model adeptly
analyzes and prioritizes multi-modal data, offering a more refined
understanding of the environment.

  * •

Our empirical evaluation demonstrates that our model surpasses existing state-
of-the-art methods in performance, particularly on the challenging Talk2Car
dataset. This includes scenarios like low-light conditions, long-text
commands, rainy weather, and dense urban environments. Our model achieves this
with less training time, superior performance, and enhanced interpretability.

This paper is structured in the following manner to facilitate a comprehensive
understanding: Section 2 offers a detailed review of the existing literature
and studies pertinent to our research area. In Section 3, we articulate the
research task at hand and present a detailed diagram of our model’s
architecture. Section 4 elaborates on the methodology of our experimental
setup, followed by an in-depth discussion of the results obtained. In Section
5, we investigate the utility and user satisfaction of our proposed model
using a questionnaire. Finally, Section 6 synthesizes the main findings of
this study and proposes potential directions for future investigations in this
field.

##  2 Related Work

###  2.1 Visual Grounding

Visual Grounding (VG) is a task that fundamentally involves localizing image
regions pertinent to a specific natural language command. The body of research
in this domain typically falls into two primary categories based on their
methodological frameworks: one-stage and two-stage methods.

One-Stage Methods in Visual Grounding: These methods are characterized by
their efficiency, primarily due to a streamlined processing paradigm. Unlike
the two-stage models, which initiate with proposing candidate regions followed
by object classification refinement, one-stage methods [20, 21, 22, 23, 24]
simultaneously extract image and command features in a unified pass,
predicting both object categories and their bounding boxes concurrently. For
instance, [20] introduced a guided attention mechanism to focus on the
referent’s central region, interpreting referring expressions in a single
stage without resorting to region proposals. [22] reformulated referring
expression comprehension as a correlation filtering process for precise object
localization. Moreover, [24] integrated a visual-linguistic verification
module within a unified one-stage framework to effectively identify regions
corresponding with descriptions. The elimination of the region proposal phase
in single-stage methods facilitates faster inference times but often requires
extended training durations and may involve trade-offs in performance.
Particularly, these methods may struggle in scenarios with densely packed
objects or overlapping regions of interest, posing accuracy challenges and
limiting their robust application in AV contexts.

Two-Stage Methods in Visual Grounding: In contrast, two-stage methods are
renowned for their meticulous object identification process within images.
These methods typically employ pre-trained models as object detectors in the
first stage to extract region-specific features [25, 26, 27, 28]. For example,
MAttNet [29] and Volta [30] utilize Faster R-CNN [31] and ResNet-101 [32] as
their backbone networks to capture region features. These features form the
basis for defining the spatial locations of potential objects of interest,
termed region proposals. In the second stage, these proposals are integrated
with linguistic features and analyzed by the framework’s decoder, which
selects regions that align closely with the linguistic input, ensuring a
semantically coherent and contextually appropriate detection result.

While two-stage methods have demonstrated efficacy in VG tasks, they are often
heavily reliant on the accuracy of pre-trained object detection models. This
reliance can be a drawback, especially when issues in bounding box
localization occur or when crucial semantic information outside the proposed
regions is overlooked. Our study adopts a two-stage approach but extends its
focus beyond traditional bounding boxes. We aim to capture a broader spectrum
of contextual information within various traffic scenarios. This approach
seeks to leverage the synergies between bounding box localization and
contextual cues, enhancing object localization and promoting context-aware
recognition in AV applications.

###  2.2 Language-to-Image in Autonomous Driving

The fusion of object detection with semantic information processing has
catalyzed significant advancements in AVs, particularly in the realm of
language grounding. A seminal work in this area is the Talk2Car dataset
introduced by Deruyttere et al. [33]. This innovative dataset, comprising
natural language commands specifically tailored for self-driving cars, serves
as a critical benchmark and sheds light on the challenges of object referral
in visual grounding (VG) for AVs.

Progressing from the foundations established by the Talk2Car dataset, recent
research has further advanced the field. Mittal’s AttnGrounder [8], for
instance, is an end-to-end model with a Cross-Modal attention module that
correlates words in a query to regions in visual scenes. Rufus et al. [10]
contributed a minimalist yet effective approach to VG in AVs. Further
enriching this domain, Deruyttere et al. [34, 3] and Rufus et al. [35]
explored uncertainties in commands and navigable region identification. Chan
et al. [18] and Cheng et al. [36] have also made significant strides, with
Cheng et al. introducing LiDAR Grounding. Dong et al. [37] presented the HuBo-
VLM model, showcasing a unified vision-language approach for human-robot
interactions.

###  2.3 Emotion Classification

Emotion classification has been a focus in several studies, including those
analyzing emotions expressed in specific events [38, 39]. Devlin et al.’s BERT
[40] integrated transformer encoders for pretraining models on NLP tasks,
including emotion classification. Yang et al. [41] introduced EmotionX-KU, a
classifier for detecting emotions in dialogue, as part of the EmotionX 2019
challenge. Sentiment GPT [42] applied GPT models to emotion classification,
highlighting their effectiveness in sentiment analysis. In our study, we
utilize GPT-4 for analyzing emotional content in commands, drawing on its
extensive pre-training to grasp subtle linguistic patterns and contextual
nuances.

###  2.4 Vision-and-Language Model

The domain of Vision-and-Language pre-trained models is characterized by two
principal architectures: single-stream and dual-stream. Single-stream models
like ImageBert [43] seamlessly integrate visual and linguistic modalities
across encoder layers for holistic data processing. In contrast, dual-stream
architectures, such as LXMERT [26] and VilBERT [44], begin with intra-modal
interactions, gradually merging modalities as processing progresses. LXMERT,
for example, employs an object-relationship encoding module to align complex
visual patterns with linguistic cues. In our research, we adopt a hybrid
approach that harnesses the strengths of both architectures, optimizing the
CAVG model for intricate tasks in AVs.

Figure 2: Schematic of the Model Architecture. The Text Encoder and the
Emotion Encoder generate a text vector and an emotion vector, respectively,
from the given command, while the Vision Encoder divides the input image into
$N$ RoIs, each represented by a vision vector. These vectors are contextually
enriched by a context encoder and then merged by a Cross-Modal Encoder using
multi-head cross-modal attention. The multimodal decoder calculates likelihood
scores for each region and selects the top-$k$ regions that best match the
semantics of the command. The final prediction is based on this fusion.

##  3 Proposed Model

###  3.1 Overview

Our aim is to devise a method that can process a frontal-view image of a
vehicle, denoted as $I$, in tandem with a natural language command, denoted as
$C$. The ultimate objective is to accurately pinpoint the destination region
within the image where the vehicle should navigate in order to comply with the
given command. To formalize this, we cast the task as a mapping problem:
identifying the most suitable Regions of Interest (RoIs) in the image $I$ that
corresponds to the destination the vehicle must reach based on the natural
language command $C$. This mapping is not trivial and involves determining the
best-matching bounding boxes within the image that are in line with the
semantic intent of the command.

As illustrated in Fig. 2, our proposed framework utilizes an encoder-decoder
architecture that comprises five specialized encoders: the Text Encoder, the
Emotion Encoder, the Vision Encoder, the Context Encoder, and the Cross-Modal
Encoder. These are integrated with a Multimodal Decoder to perform the task of
destination identification.

###  3.2 Text Encoder

The Text Encoder is primarily responsible for processing the natural language
command $C$. To achieve this, we use the well-known Bidirectional Transformer
(BERT) architecture [40]. BERT has emerged as a seminal advancement in NLP due
to its remarkable capabilities in capturing intricate contextual nuances and
relationships within textual data. From a technical perspective, BERT uses a
multi-layer bidirectional transformer architecture with a deep stack of
transformer encoder layers. These layers include self-attention mechanisms
that allow the model to capture dependencies between words in both forward and
backward directions, ensuring a thorough contextual understanding of the
entire input sequence.

Specifically, the Text Encoder uses BERT’s robust pre-trained language
representations to generate a comprehensive text vector, denoted
$\bm{O}_{\textit{text}}$. The input command $C$ is tokenized into a sequence
using BERT’s WordPieces tokenizer and then fed into the BERT model. The
resulting output is a unified command embedding
$\\{w_{1},w_{2},w_{3},\ldots,w_{n}\\}$ that serves as
$\bm{O}_{\textit{text}}$. This embedding encapsulates the textual
characteristics of the input command, including its meaning and context. It
lays the foundation for subsequent multimodal fusion and contextual analysis,
enabling CAVG to seamlessly integrate and process language commands in the
realm of autonomous driving. Formally,

$\bm{O}_{\textit{text}}=\phi_{\textit{BERT}}\,\left(\phi_{\textit{Tokenizer}}\,(C)\right)=\\{w_{1},w_{2},w_{3},\ldots,w_{n}\\}$
(1)

where $\phi_{\textit{Tokenizer}}\,(C)$ represents the transformation of the
input command $C$ into its corresponding token embeddings using the BERT’s
WordPiece tokenizer, while the $\phi_{\textit{BERT}}\,(\cdot)$ denotes the
BERT architecture. In particular, our Text Encoder is more than just an off-
the-shelf implementation of BERT. Instead, we are fine-tuning and adapting
BERT to the specific requirements and nuances of language used in the context
of autonomous driving. This fine-tuning process adjusts the model to better
understand and interpret the unique vocabulary, syntax, and command structures
encountered in the context of autonomous driving.

###  3.3 Emotion Encoder

The Emotion Encoder in our study is crucial for interpreting emotions in
commander commands, which is vital for the safe and efficient operation of
AVs. This encoder is adept at classifying human emotions, drawing on the
advanced capabilities of Large Language Models (LLMs), specifically GPT-4, as
referenced by [45]. GPT-4’s pre-training on a broad corpus of text data equips
it with a deep understanding of complex linguistic patterns, semantics, and
contextual nuances. Its core self-attention mechanism enables nuanced emotion
analysis, allowing it to assign different levels of importance to various
words in a sentence.

The Emotion Encoder processes commander commands and categorizes them into
three distinct emotional states:

  * •

Urgent: These commands require immediate action due to their time-sensitive or
critical nature. For example, a commander yelling “Wow hold on! That looks
like my stolen bike over there! Drop me off next to it.”, conveys a sense of
urgency that requires immediate attention. The Emotion Encoder recognizes this
emotional state and interprets this command as requiring immediate attention.
Consequently, the destination is inferred as the closest and safest place to
pull over, in line with the commander’s anxious emotional state.

  * •

Commanding: These commands are characterized by their direct and unambiguous
nature, specifying what actions the vehicle should take without an inherent
time sensitivity. For instance, a command like, “Make a left turn at the next
intersection.”, is clear and commanding. While the command itself is
straightforward, our Emotion Encoder detects the absence of urgency in the
commander’s tone. It categorizes this as a “commanding” instruction. In
response, the model processes the command accordingly, planning a left turn at
the next intersection without the need for immediate action.

  * •

Informative: Neutral instructions that convey useful information without a
strong emotional tone. Informative commands are primarily characterized by
their objective nature, focusing on providing factual details about a
particular subject or object. These commands tend to be descriptive, where the
content predominantly revolves around elucidating specific attributes or
features of an entity without conveying a pressing need for action.

The Emotion Encoder utilizes the GPT-4 API for emotion analysis. The API setup
involves secure communication with OpenAI’s servers, and each commander
command is preprocessed to align with GPT-4’s input requirements. The command
is sent to GPT-4, which then analyzes the text and returns a response
indicating the emotional tone. This response is post-processed to categorize
the command into one of the predefined categories.

The output of the Emotion Encoder $\bm{O}_{\textit{emo}}$ is the predicted
emotion category $e$ for the input command $C$, which can be defined as
follows:

$\bm{O}_{\textit{emo}}=\phi_{\textit{GPT}}\,(C)=\\{e\\}$ (2)

where the $\phi_{\textit{GPT}}\,(\cdot)$ represents the GPT-4 framework.
Subsequently, the emotion vector $\bm{O}_{\textit{emo}}$ is fused with the
text vector $\bm{O}_{\textit{text}}$ to incorporate the emotional context.
This integration is devised to indicate CAVG towards providing human-like
responses that resonate with both the given command and the emotional
disposition of the commander.

Figure 3: The Output of the Vision Encoder on the Talk2Car Dataset. Each
bounding box is labeled with an attribute class followed by an object class.
Ground truth bounding boxes are depicted in blue, while the output bounding
boxes of CAVG are highlighted in green. A natural language command associated
with each visual scenario is also displayed below the image for context.

###  3.4 Vision Encoder

The Vision Encoder is architected to extract rich visual information from the
given view image $I$ for subsequent multimodal analysis. Within this encoder,
the CenterNet [46] model is utilized to identify and localize entities and
elements within the observed scene in the raw image. Specifically, CenterNet
is built on the foundation of two basic components: Fast R-CNN [31] and
ResNet-101 [32]. Known for its efficiency and accuracy, Fast R-CNN utilizes a
region proposal network (RPN) to generate region proposals. These proposals
are then subjected to a two-stage process involving a classifier and a
regressor. The classifier is responsible for object detection, while the
regressor refines the bounding box coordinates. As shown in Fig. 3, through
this process, CAVG can effectively identify prominent entities and detect
their classes in the input image, such as trees, vehicles, bicycles, and
pedestrians. In particular, these identified entities are encapsulated as
RoIs, rich in semantic features and spatial correlations among different
entities.

Complementing the capabilities of Fast R-CNN, ResNet-101 is introduced into
the framework to enhance feature representation, with a keen focus on
capturing finer details, textures, and distinguishing attributes of detected
objects. Such nuances are essential for understanding commands that may refer
to specific visual cues or unique scene features. For example, given a visual
scene with the command “Could you please park before the red car?”, the
ResNet-101 is adept at capturing subtle color variations and textures within
the RoIs and embedding them into high-dimensional visual representations. This
empowers CAVG to pinpoint the relevant vehicles and greatly helps to provide
accurate, context-aware responses.

Overall, this synergistic integration within the CenterNet model enables the
encoder to produce the high-dimensional visual representations, encapsulated
in a vision vector, labeled $\bm{O}_{\textit{vision}}$, which can be given as:

$\bm{O}_{\textit{vision}}=\phi_{\textit{CenterNet}}\,(C)=\\{b_{1},b_{2},b_{3},\ldots,b_{n}\\}$
(3)

Here, $\phi_{\textit{CenterNet}}\,(\cdot)$ represents the CenterNet model and
$\\{b_{1},b_{2},b_{3},\ldots,b_{n}\\}$ is the processed RoIs in the vision
vector.

Figure 4: Raw Images and Examples Output from Context Encoder. CAVG
automatically learns to perform object segmentation without a complex self-
supervised training recipe or any fine-tuning with segmentation-related
annotations. For each image set, we visualize the raw image on the left and
the self-attention maps of the image in different layers of the encoder on the
right.

###  3.5 Context Encoder

Our model confronts a more intricate challenge than traditional Referring
Image Segmentation tasks due to its multimodal nature and the unstructured
nature of its ground truth. In the “talk-to-car" scenario, the fusion of
unrestricted linguistic expressions with diverse visual scenes poses several
challenges. These include co-reference resolution, precise identification of
named entities, understanding complex object relationships, accurate
association of attributes with corresponding visual elements, and pinpointing
the object of interest within a command.

Diverging from existing methods that primarily focus RoIs [8, 47, 9], our
Context Encoder adopts a more holistic approach. It is designed not only to
identify key focal points within the input image but also to discern the
broader contextual relationships within the entire visual scene, going beyond
the limitations of bounding boxes. To achieve this, we utilize the Vision
Transformer (ViT) and the state-of-the-art Vision-and-Language (V&L) pre-
training model, and LLM Bootstrapping Language-Image Pre-training Model (BLIP)
[48]. ViT, in particular, revolutionizes the comprehension capabilities of the
Context-Aware Visual Grounding (CAVG) model with its unique ability to
consider the entire visual scene as a unified entity. Unlike conventional
object-centric methods reliant on bounding boxes, ViT offers a comprehensive
perspective of the visual scene. As depicted in Fig. 4, ViT captures essential
contextual details—such as lane markings, pedestrian paths, traffic signs, and
other nuances—that are often missed by rigid bounding box constraints. This
expanded scope enables our model to assimilate a wider range of visual
information and contextual cues, ensuring a thorough semantic interpretation.

In concert with ViT, BLIP acts as a bridge between textual and visual
information, enabling CAVG to extract cross-modal features and make meaningful
connections between the words in the command and the semantic elements present
in the image. This fusion of linguistic context with rich visual cues
facilitates a more holistic and nuanced understanding of the scene, enhancing
our model’s ability to accurately ground textual commands in the visual
scenes. The Context Encoder ingests the context vector
$\bm{O}_{\textit{context}}$ from text vector $\bm{O}_{\textit{text}}$ and the
input image $I$, which can be represented as follows:

$\bm{O}_{\textit{context}}=\phi_{\textit{BLIP}}\,\left(\phi_{\textit{ViT}}\,(I)\|\bm{O}_{\textit{text}}\right)$
(4)

where $\phi_{\textit{BLIP}}$, and $\phi_{\textit{ViT}}$ are the BLIP and ViT
frameworks, respectively. In addition, $\|$ denotes matrix concatenation and
the context vector $\bm{O}_{\textit{context}}$ not only augments the model
with additional visual cues but also broadens its understanding of the scene.
This, in turn, enhances the model’s ability to make more accurate predictions.

###  3.6 Cross-Modal Encoder

This encoder synergizes the output of the previous encoders through a multi-
head cross-modal attention mechanism. Unlike conventional methods like
MaskFormer [49], DETR [50], and MDETR [51], which are computationally
expensive, our approach uses a “hindsight fusion" strategy. This enables more
efficient processing by harmoniously merging visual and text vectors before
they are inputted into the Multimodal Decoder.

Figure 5: Overall Framework of Multi-head Cross-modal Self-attention
Mechanism.

As depicted in Fig. 5, the encoder projects the text vector
$\bm{O}_{\textit{text}}$ and the emotion vector $\bm{O}_{\textit{emo}}$ into
query representation $Q$ using Multilayer Perceptrons (MLP). Similarly, the
vision $\bm{O}_{\textit{vision}}$ and context $\bm{O}_{\textit{context}}$
vectors are converted into the key $K$ and standardized value $V$
representations, respectively. To further enhance the text and vision vectors,
we append extra tokens $L_{q}$ and $L_{k}$ to the end of the input text vector
$\bm{O}_{\textit{text}}$ and vision vector $\bm{O}_{\textit{vision}}$,
respectively. Then, these representations are utilized to compute a
comprehensive fused vector $\alpha$, as outlined in the equations. In
addition, it is worth noting that our cross-modal attention mechanism draws
inspiration from the original Transformer [52] architecture and can also be
extended to incorporate multiple attention heads, with a residual connection
applied after these attention heads [53]. Formally,

$\left\\{\begin{array}[]{l}{Q}_{i}={W}^{Q}(\phi_{\textit{Linear}}(\bm{O}_{\textit{vision}}+L_{q}))\\\
{K}_{i}={W}^{K}(\phi_{\textit{Linear}}((\bm{O}_{\textit{emo}}\|\bm{O}_{\textit{text}})+L_{k}))\\\
{V}_{i}={W}^{V}\bm{O}_{\textit{context}}\end{array}\right.$ (5)

where ${W}^{Q},{W}^{K},{W}^{V}$ denote the learnable weights, and
$\phi_{\textit{Linear}}$ represents the fully connected layer. These tokens
$L_{q}$ and $L_{k}$ are generated as position embeddings from the given
command $C$ and the raw image $I$ by leveraging the single-stream architecture
UNITER (Universal Image-TExt Representation) [54], which can be expressed as
follows:

$\left\\{\begin{array}[]{l}L_{q}=\phi_{\textit{UNITER}}\,\left({C}\right)\\\
L_{k}=\phi_{\textit{UNITER}}\,\left({I}\right)\end{array}\right.$ (6)

where $\phi_{\textit{UNITER}}$ denotes the UNITER architecture. Moreover, the
$i$th self-attention head $\textit{head}_{i}$ and the output of the multi-head
cross-modal self-attention mechanism ${\alpha}$ can be defined mathematically
as:

$\displaystyle{\alpha}$
$\displaystyle=\sum_{i=1}^{h}{\textit{head}_{i}}+\phi_{\text{Linear}}(\bm{O}_{\textit{vision}}+L_{q})$
(7)
$\displaystyle=\sum_{i=1}^{h}{\phi_{\textit{softmax}}}\left(\frac{{Q}_{i}({K}_{i})^{\top}}{\sqrt{d_{k}}}\right){V}_{i}+\phi_{\textit{Linear}}(\bm{O}_{\textit{vision}}+L_{q})$

where $h$ is the total number of attention heads,
${\phi_{\textit{softmax}}}(\cdot)$ is the softmax activation function, and
${d}_{k}$ represents the dimensionality of the projected key vectors.
Furthermore, the output of this attention mechanism ${\alpha}$ is then
embedded by the UNITER to produce the fused cross-modal vector
${\bar{\alpha}}$, which can be given as follows:

${\bar{\alpha}}=\phi_{\textit{UNITER}}\,\left({\alpha}\right)$ (8)

Our methodology prominently features the UNITER architecture, renowned for its
exceptional capacity to align and integrate visual and textual data into a
unified representation. Built upon the transformer model, UNITER processes
images and text in parallel, transforming images into patch embeddings
analogous to text tokens. This method facilitates a seamless integration of
linguistic and visual modalities, a crucial aspect for interpreting complex
multimodal data in our study. Pretrained on a broad spectrum of V&L tasks,
UNITER can discern a wide range of visual and textual patterns. This extensive
pre-training is especially beneficial for our application, where interpreting
the intricate interplay between natural language commands and corresponding
visual cues is critical. UNITER’s ability to create a joint representation of
text and images is key to effectively grounding language commands within the
specific visual contexts encountered by AVs.

###  3.7 Multimodal Decoder

The final stage of our architecture is the Multimodal Decoder, which plays a
pivotal role in synthesizing and interpreting the multimodal data. This
decoder integrates the high-level vector ${L_{q}}$, embedded by the UNITER,
with the fused cross-modal vector ${\bar{\alpha}}$. Its primary function is to
discern and rank the top-$k$ RoIs in the image, thereby identifying the most
relevant target region for the given command. Comprising $m$ layers, the
Multimodal Decoder is uniquely structured. Each layer is an amalgamation of a
Transformer Decoder [52] and a Region-Specific Dynamic (RSD) Layer Attention
Mechanism. The RSD mechanism, inspired by Chan et al. [18], represents an
advanced evolution in attention mechanisms, designed specifically to address
the multimodal complexity of our task.

Within each layer, the Transformer Decoder generates a high-dimensional hidden
state from its input. This state is then processed by the RSD Layer, a key
component that enhances the model’s interpretative capabilities. The RSD
mechanism dynamically assigns attention weights to the representation vectors
of a region across all encoder layers, rather than relying solely on the top
layer representation. This approach allows for a more nuanced aggregation of
features, ranging from concrete to abstract, across different layers of the
encoder. For each region proposal $v_{i}$, the RSD Layer evaluates its
representation $h_{l}^{v_{i}}$ at each encoder layer $l$, computing relevance
scores $\alpha_{il}$. These scores are used to aggregate the representation
vectors across layers, enabling the model to consider a wide array of features
for each region and determine its relevance to the command. The hidden states
from all layers, each weighted by their computed relevance, are fused into a
singular representation. This combined output is then passed through a
Multilayer Perceptron (MLP) followed by a softmax activation function,
resulting in multiple hypothesis outputs with associated credibility scores.
The bounding box with the highest credibility score is selected as the final
output. To further enhance feature retention and extraction, skip connections
akin to those in ResNet [32] are employed. These connections feed the high-
level vector ${L_{q}}$ into every layer of the Multimodal Decoder. This design
allows the model to eliminate redundant features while preserving those
essential for accurate interpretation and decision-making, thereby improving
the accuracy and reliability of the model’s output.

##  4 Experiments

###  4.1 Dataset

We assess CAVG using the Talk2Car [33] dataset, a derivative of the NuScenes
[55] dataset, featuring 11,959 natural language commands across 9,217 images
of urban environments from Singapore and Boston. The data set is particularly
challenging and diverse, captured under various conditions including different
times of the day and weather scenarios. It poses intricate tasks like object
disambiguation and understanding elaborate text queries, with commands
averaging 11 words in length, fitting CAVG’s data requirements. Examples
include complex instructions like “This car just passed a crosswalk, slow down
to check for pedestrians!”. We utilize it as a benchmark to assess CAVG’s
accuracy in this study.

Figure 6: Statistics of Talk2car Datasets on (a) Word Cloud and (b)
Distribution of Referred Object Number.

###  4.2 Evaluation Metric

In line with the C4AV challenge, we use the ${IoU}_{0.5}$ score as the
evaluation metric. It is alternatively referred to as the ${AP}_{50}$ score,
which is defined as the percentage of Intersection over Union (IoU) score
values that exceed the 0.5 thresholds with the ground truth bounding boxes.
This evaluation metric quantifies is widely used in challenges and benchmarks
such as the PASCAL [56], VOC [57], and COCO [58] datasets, provides a rigorous
measure of the accuracy of our predictions, and is consistent with established
practices in computer vision and object recognition tasks. Let $A$ and $B$
denote the predicted bounding box and the ground truth bounding box,
respectively. In this study, each bounding box is represented by the
coordinates of the positions of its upper-left $(x_{1},y_{1})$ and lower-right
$(x_{2},y_{2})$ corners. The following equation elaborates on the computation
of IoU between the predicted bounding box $A$ and the ground truth bounding
box $B$:

$\text{IoU}(A,B)=\frac{\text{Area of Overlap}\,(A,B)}{\text{Area of
Union}\,(A,B)}$ (9)

###  4.3 Training and Implementation Details

Our model is trained to converge on an NVIDIA A100 40GB GPU for approximately
an hour. It is trained for 6 epochs using the AdamW optimizer, with a batch
size of 16 and an initial learning rate set at $10^{-4}$. In addition, the
loss function for CAVG is the binary cross-entropy (BCE) loss, with scheduling
regulated by the Cosine Annealing Warm Restarts [59] strategy. Further
specifics regarding the implementation and essential parameter settings of
CAVG are provided as follows:

Text Encoder. In the text encoder, we use a pre-trained BERT model to generate
text embeddings, enforcing a maximum sentence length of 60 tokens. We
configure the BERT encoder with 16 hidden layers and an embedding layer with a
vocabulary size of 30,524. The epsilon value for the layer normalization is
set to 1e-12. The hidden size within the text encoder is set to $d=768$,
resulting in a text vector labeled $\bm{O}_{\textit{text}}$ with a dimension
of $d$.

Emotion Encoder. In this encoder, GPT-4 is used as an emotion classifier to
distinguish emotion categories and generate an embedding
$\bm{O}_{\textit{emo}}$ with a dimension of 768, ensuring semantic congruence
with $\bm{O}_{\textit{text}}$. We then concatenate $\bm{O}_{\textit{emo}}$ and
$\bm{O}_{\textit{text}}$, projecting them to a dimension of $d$.

Vision Encoder. We resize the input image from 1600$\times$900 to
800$\times$450 in the Vision Encoder. Then, Fast R-CNN and ResNet-101 are used
as the backbone to locate the bounding boxes of all the objects, and the
region proposal threshold is set to 36. Within each region proposal, image
information is extracted from the image feature $\bm{O}_{\textit{vision}}$
with the dimension of 36$\times$1024.

Context Encoder. The Context Encoder utilizes the complete raw image $I$ in
conjunction with the text vector $\bm{O}_{\textit{text}}$. Within this
framework, we choose an encoder between Vision Transformer-Base and Vision
Transformer-Large for the vision embedding layer. Specifically, the ViT-B is
configured with a depth of 12, a vision width of 768, a patch size of 16, a
ratio of MLP hidden dimension to embedding dimension of 4, and a multi-head
attention mechanism with 12 heads. The resulting encoder output is the context
embedding $\bm{O}_{\textit{context}}$ with a dimension of $d$.

Cross-Modal Encoder. The vectors of text $\bm{O}_{\textit{text}}$, emotion
$\bm{O}_{\textit{emotion}}$, vision $\bm{O}_{\textit{vision}}$, and context
$\bm{O}_{\textit{context}}$ vectors are fed into this encoder to be converted
by the attention mechanism into representations of key, query and values, each
with a dimension of 1024. The encoder is configured with 16 heads to optimize
the attention mechanism. The UNITER then embeds both ${\alpha}$ and the text
feature, using a hidden size of 1024 and producing an output of dimension $d$.

Multimodal Decoder. The Multimodal Decoder is designed with 12 attention
layers. Within each layer, we utilize a multi-head attention mechanism
configured with 12 heads, and both the vision feature and the text image have
a hidden size set to $d$. Additionally, every attention layer incorporates a
linear forward progression followed by batch normalization and dropout layers.

###  4.4 Experiment Setup

Our experimental protocol aligns with the Commands For Autonomous Vehicles
(C4AV) Challenge protocol. We split the Talk2car dataset into 8,349 training,
1,163 validation, and 2,447 test samples, which constitute 69.8%, 9.7%, and
20.5% of the total dataset. We refer to the complete testset as the full
testset. With this configuration in mind, we compute the distribution of the
word counts for the natural language command on the Talk2Car dataset. The
statistical results, shown in Fig. 7, indicate that the training, validation,
and testsets exhibit similar data distribution profiles for input commands.

Figure 7: Word Count Distribution of Commands in the Training, Validation, and
Test Set.

To rigorously evaluate the effectiveness of our model in real-world scenarios,
we meticulously partitioned the validation set based on the complexity of
linguistic commands and the inherent challenges of visual environments. It is
worth noting that longer commands may introduce irrelevant details or be more
complex for AVs to understand. In addition, specific visual scenarios - such
as low-light night scenes, congested urban environments with complex object
interactions, ambiguous command prompts, and rainy days where visibility is
compromised - invariably complicate prediction. To this end, we prudently
select more than 23 words in length and categorize the aforementioned
challenging scenarios into specialized subsets, designated as the Long-text
and Corner-case testsets, respectively. This categorization allows for more
focused evaluations.

In addition, the Corner-case testset consists of three main subsets: the
scenarios with restricted vision, the scenarios with multi-agent interactions,
and the scenarios with ambiguous command prompts, containing 65, 75, and 85
samples, respectively. Figure 8 displays the example scenes for each subset;
it is pertinent to mention that a single sample can simultaneously span
multiple subsets, provided it is consistent with the definition of multiple
scenarios. Correspondingly, we treat the C4AV test set as a “normal” subset
for comparative analysis. Furthermore, for the Long-text testset, we employed
a data augmentation strategy to increase the richness of the dataset without
deviating from the original semantic intent. Using LLM GPT-3.5, we extended
certain commands, resulting in commands ranging from 23 to 50 words. This
allows us to further evaluate the model’s ability to process extended
linguistic input, ensuring a comprehensive assessment of our model’s
adaptability and robustness.

Table 1: Performance Comparison of Our Model with Various SOTA Methods on the
Talk2Car Dataset using $IoU_{0.5}$ Metric. Models are categorized into three
types: one-stage, two-stage, and others, and evaluated based on architectural
backbones: Vision for visual feature extraction, Language for semantic
information extraction, and Global for holistic data assimilation. Additional
components evaluated include emotion classification (EmoClf.), global image
feature extraction (Global Img Repr.), linguistic augmentation (NLP Augm.),
and visual augmentation (Vis Augm.). ‘Yes’ indicates the use of a feature,
‘No’ its absence, and ‘-’ represents undisclosed data. This categorization
elucidates the underlying components and strategies influencing each model’s
performance. Bold and underlined values represent the best and second-best
performance, respectively. For comprehensive results, visit the Official
Leaderboard.

Model | $IoU_{0.5}$ | Backbones | Methods | Augmentation
---|---|---|---|---
Vision | Language | Global | EmoClf. | Global Img Repr. | NLP Augm. | Vis Augm.
One-stage: |  |  |  |  |  |  |  |
MAC | 50.5 | ResNet-101 | LSTM | ResNet+CNN | No | Yes | No | Yes
AttnGrounder | 63.3 | ResNet-53 | LSTM | Darknet-53 [60] | No | Yes | No | Yes
Two-stage: |  |  |  |  |  |  |  |
OSM | 35.3 | ResNet-18 | GRU | ResNet-18 | No | Yes | - | -
SCRC | 43.8 | ResNet-18 | GRU | ResNet-18 | No | Yes | - | -
Bi-Dir.retr | 44.1 | ResNet-18 | LSTM | No | No | No | No | No
MSRR (Top-16) | 59.9 | ResNet-101 | LSTM | No | No | No | No | No
MSRR | 60.1 | ResNet-101 | LSTM | No | No | No | No | No
ASSMR (4th) | 66.4 | ResNet-18 | GRU | ResNet-101 | No | Optional | Yes | Yes
CMSVG (3rd) | 68.6 | EfficientNet [61] | Sent.-Transf [62] | No | No | No | No | No
CMRT (2nd) | 69.1 | ResNet-152 | Transformer [52] | ResNet-152 | No | Yes | - | -
Sentence-BET | 70.1 | ResNet-152 | Sent.-Transf [62] | ResNet-152 | No | Yes | - | -
Stacked VL-BERT (1st) | 71.0 | ResNet-101 | VL-BERT | ResNet-101 | No | Yes | - | -
Others: |  |  |  |  |  |  |  |
C4AV-Base | 44.3 | - | - | - | No | - | No | -
A-AAR | 45.1 | VGG-16[63] | Hier.-LSTM [64] | - | No | Yes | - | -
MaskObjs (Top-16) | 54.3 | - | - | - | No | - | - | -
VL-BERT (base) | 68.3 | - | - | - | No | - | - | -
SOTA (Top 1-6) | 66.4±5.6 |  |  |  |  |  |  |
SOTA (Top 7-12) | 50.1±6.8 |  |  |  |  |  |  |
CAVG (50%) | 70.3 | CenterNet | BERT/BLIP | UNITER/BLIP | Yes | Yes | Yes | Yes
CAVG (75%) | 72.1 | CenterNet | BERT/BLIP | UNITER/BLIP | Yes | Yes | Yes | Yes
CAVG | 74.6 | CenterNet | BERT/BLIP | UNITER/BLIP | Yes | Yes | Yes | Yes

Figure 8: Example for the Subsets of the Corner-case Test Set.

####  4.4.1 Performance Benchmarking Against SOTA Models on full Testset

In our research, we rigorously evaluated CAVG’s performance against a
comprehensive array of state-of-the-art (SOTA) methods. These include well-
known benchmarks such as STACK [47], OSC [65], SCRC [65], C4AV-Base [66],
A-ARR [67], MAC [66], MaskObjs (Top-16) [9], MSRR [9], AttnGrounder [8], VL-
BERT (Base) [65], and Sentence-BET [17]. Additionally, we included the top
four contenders from the prestigious C4AV Challenge [33]—namely, Stacked VL-
BERT (1st) [68], CMRT (2nd) [69], CMSVG (3rd) [10], and ASSMR (4th) [70]—to
ensure a holistic assessment. The complete performance metrics for this
comparative analysis are available in Table 1.

Remarkably, within the full testset, our model did not just hold its own—it
dominated across the board. Not only did it surpass all the aforementioned
SOTA methods, but it also eclipsed the performances of all models from the
C4AV Challenge leaderboard. In terms of the ${IoU}_{0.5}$ metric, the
superiority of CAVG is clear: it outperformed Stacked VL-BERT (1st) by a
margin of 4.8%, outperformed CMRT (2nd) by 7.4%, eclipsed CMSVG (3rd) by 7.9%,
and dramatically outperformed ASSMR (4th) by an impressive 11.0%. This
compelling empirical evidence strongly attests to the exceptional efficacy of
our proposed model in the task at hand.

####  4.4.2 Performance Evaluation of Training on Reduced Datasets

In order to showcase the scalability and efficiency of CAVG, we conducted an
experiment in which the model was trained on different subsets of the training
set, specifically 50% and 75%. As illustrated in Tables 1, under the labels
CAVG (50%) and CAVG (75%), CAVG exhibits commendable performance. When
conditioned on the 75% subset of the training data, CAVG consistently
outperforms all other baseline models, outperforming C4AV SOTA baselines
Stacked VL-BERT (1st) and CMRT (2nd) by 5.07% and 7.96%, respectively.
Impressively, an interesting finding is that when the model is trained on only
50% of the training data, CAVG displays an admirable performance by achieving
significantly lower ${IoU}_{0.5}$ values compared to most of the baselines,
and outperforms almost all of the top 1-6 SOTA baselines. These results
underscore the potential of CAVG to provide competitive performance even in
the absence of extensive training data. This property is of paramount
importance, as it suggests potential avenues for economizing data acquisition
and annotation efforts, particularly relevant for the autonomous vehicle
industry, where data acquisition can be both time-consuming and costly.

Figure 9: Evaluation Results in Long-text and Corner-case Testsets.

####  4.4.3 Performance Evaluation on Long-text and Corner-case Testsets

For a holistic insight into the robustness and adaptability of CAVG, we
performed further evaluations on the Long-text and Corner-case testsets, as
detailed in Table 2 and Table 3, respectively. To our knowledge, few methods
in this area provide the code for these proposed models. Therefore, our
comparison focuses on competing open-source models. When confronted with the
challenges of the Long-text testset, as shown in Fig. 9 (a), CAVG consistently
surpasses all other baselines, showing an improvement of 4.0%-10.1% over the
SOTA baselines. Notably, even our model variant trained on only 75% of the
dataset, CAVG (75%), shows commendable performance, rivaling if not surpassing
the current SOTA model, Stacked VL-BERT (1st). Furthermore, the iteration of
CAVG trained with only half the dataset, CAVG (50%), shows impressive
performance, outperforming the majority of the established models.
Interestingly, our models - CAVG and CAVG (75%) - excel in multi-agent
scenarios compared to normal ones when provided with sufficient training data.
These results highlight their superior ability to understand and interpret
commands of considerable length and complexity.

Shifting our gaze to the Corner-case testset in Fig. 9 (b), the strengths of
CAVG become even more apparent in the challenging scenes. It exceeds current
SOTA baselines, with accuracy gains ranging from 1.9%-4.9% on the Talk2car
dataset. Moreover, even the CAVG trained on only 75% of the dataset
outperforms other SOTA baselines, while its counterpart, the CAVG trained on
only 50% of the dataset, still manifests commendable performance. This
demonstrates the inherent robustness of CAVG, even under challenging
conditions, and underscores its promise for use in real-world AV scenarios.

Table 2: Performance Comparison of Our Model with Open-source SOTA Methods on
the Long-text Testset using $IoU_{0.5}$ Metric. Bold and underlined values
represent the best and second-best performance, respectively.

Model | Top-k Longest Comments
---|---
400 | 350 | 300 | 250 | 200 | 150 | 100 | 50
AttnGrounder | 61.9 | 60.5 | 59.8 | 58.7 | 58.6 | 57.8 | 57.0 | 56.1
CMSVG | 66.5 | 65.9 | 65.3 | 64.3 | 64.1 | 63.8 | 63.7 | 63.5
Stacked VL-BERT (1st) | 67.5 | 70.9 | 70.3 | 67.3 | 67.1 | 66.9 | 66.4 | 66.2
CAVG (50%) | 63.5 | 64.0 | 64.3 | 63.1 | 61.7 | 60.1 | 56.2 | 55.2
CAVG (75%) | 66.3 | 70.4 | 69.4 | 68.9 | 67.4 | 65.6 | 65.3 | 65.1
CAVG (100%) | 74.1 | 73.6 | 73.1 | 71.9 | 72.2 | 71.9 | 71.0 | 69.1

Table 3: Performance Comparison of Our Model with Open-source SOTA Methods on
the Corner-case Testset using $IoU_{0.5}$ Metric. Bold and underlined values
represent the best and second-best performance, respectively.

Model | Subtestset Categories
---|---
Normal | Restricted | Multi-agent | Ambiguous Command
AttnGrounder | 63.9 | 62.7 | 64.8 | 64.3
CMSVG | 68.7 | 69.3 | 73.8 | 67.8
Stacked VL-BERT (1st) | 71.1 | 70.2 | 71.8 | 69.8
CAVG (50%) | 70.4 | 68.8 | 70.4 | 69.1
CAVG (75%) | 72.1 | 71.3 | 73.1 | 70.2
CAVG (100%) | 74.6 | 72.1 | 75.2 | 72.8

####  4.4.4 Comparative Analysis of Model Inference Speed

We evaluate the inference speed of CAVG models for different architectures in
the Talk2car dataset. Specifically, we test the inference speed of all models
on a single NVIDIA GeForce RTX 3090 24GB GPU for a fair comparison. Table 4
presents a comparative analysis of various models against the CAVG (baseline),
which employs the standard R-CNN [71] as a replacement for the Vision Encoder,
omitting the BLIP and Context Encoder. These models are defined as follows,
with each compared to the full proposed model CAVG:

  * •

CAVG: The full model proposed in this paper.

  * •

CAVG (ResNet-152): Uses ResNet-152 in the Vision Encoder for visual feature
extraction.

  * •

CAVG (ResNet-50): Includes ResNet-50 in the Vision Encoder for vision vector
extraction.

  * •

CAVG (ResNet-18): Includes ResNet-18 in the Vision Encoder for vision vector
extraction.

  * •

CAVG (BLIP): Applies BLIP only in the Text and Context Encoders for vision and
context vector extraction.

  * •

CAVG (Small): A scaled-down model featuring a reduced number of attention
heads (4 heads) in the Cross-Modal Encoder with 256d hidden states, and fewer
attention layers (4 layers) in the RSD.

We obtain the average inference time of CAVG (baseline) is about 0.037s per
sample. Meanwhile, using the Vision Encoder can speed up the inference speed,
showing at least improvements of 20.1% compared to the baseline. This result
demonstrates that incorporating CenterNet into the Vision Encoder
significantly increases processing speed, while achieving better performance
in terms of prediction accuracy, underscoring its effectiveness in visual
grounding contexts for AVs where accuracy, real-time responses are critical.
Despite utilizing smaller ResNet variants in the Vision Encoder, the CAVG
(ResNet-18) and CAVG (ResNet-50) models maintain strong predictive
capabilities, outshining many SOTA models in the full test set. They also
significantly boost inference speeds by 31. 5% and 28. 3%, respectively,
compared to CAVG (baseline). Intuitively, the inference time increases with
the larger size of the ResNet used in the Vision Encoder. However, the CAVG
(ResNet-152), despite its larger size, still provides faster inference
capabilities than the CAVG (baseline). It is noted that the CAVG (Small)
achieves an optimal balance between computational efficiency and prediction
accuracy. This model is 62.9% faster in inference than the CAVG (baseline),
while also improving prediction performance by 2.1% compared to Stacked VL-
BERT (1st). This highlights the potential of the proposed model to achieve
high prediction accuracy with significantly reduced computational
requirements.

Table 4: Inference Time of CAVG Models for Different Architectures and their
Predicted Performance in the Talk2car dataset. CAVG (baseline) is selected as
the baseline for comparative analysis. These models are evaluated based on
architectural backbones: Vision for visual feature extraction, Language for
semantic information extraction, and Global for holistic data assimilation.
Bold and underlined values represent the best and second-best performance,
respectively.

Model | Accuracy | Speed | Backbones
---|---|---|---
$IoU_{0.5}$ | Accuracy-up (%) | Sample Size | Inference Time (s) | Speed-up (%) | Vision | Language | Global
CAVG (baseline) | 61.4 | - | 64 | 2.43212 | - | R-CNN | BERT | -
CAVG (ResNet-152) | 74.1 | 20.68% | 64 | 1.94372 | 20.1% | ResNet-152 | BERT/BLIP | UNITER/BLIP
CAVG (ResNet-50) | 72.2 | 17.59% | 64 | 1.74321 | 28.3% | ResNet-50 | BERT/BLIP | UNITER/BLIP
CAVG (ResNet-18) | 68.8 | 12.05% | 64 | 1.66528 | 31.5% | ResNet-18 | BERT/BLIP | UNITER/BLIP
CAVG (BLIP) | 71.4 | 16.28% | 64 | 1.77284 | 27.1% | ResNet-101 | BLIP | BLIP
CAVG (Small) | 72.5 | 18.08% | 64 | 0.90134 | 62.9% | ResNet-18 | BERT/BLIP | UNITER/BLIP
CAVG | 74.6 | 21.50% | 64 | 1.90446 | 21.7% | ResNet-101 | BERT/BLIP | UNITER/BLIP

###  4.5 Case Study for RSD Layer Attention Weights

We visualize the distribution of layer attention weights for 13 layers in a
multimodal decoder to further showcase the effectiveness of our proposed RSD
layer attention mechanism. We classify the input regions into two distinct
groups based on their $IoU$ alignment with the ground-truth region:

  * •

$IoU>0$: encompassing all regions with an $IoU$ exceeding 0, indicating an
overlap with the ground-truth region.

  * •

$IoU=0$: Constituting regions exhibiting no overlap, having an $IoU$ of
precisely 0.

Figure 10: Attention Weights over all Decoder Layers for RSD Layer Attention.
The 0-th layer indicates the embedding layer, while the 1-st denotes the first
layer of RSD layer attention, and the 12-th represents the last layer.

As shown in Fig. 10, the higher decoder layers (especially layers 7 to 10) are
endowed with a larger fraction of attention weights. This observation suggests
that vectors have a greater influence on these higher layers, possibly due to
increased cross-modal interactions. Interestingly, contrary to our initial
expectations, the top layer does not dominate the attention weights. On the
contrary, its output appears to be less central. This is a marked departure
from traditional techniques that rely predominantly on the topmost layer
representation to predict the best-aligned region, potentially sidelining the
nuanced cross-modal features inherent in other layers. In light of this, we
introduce the RSD layer mechanism, which is designed to dynamically allocate
attention weights across all decoder layers, anchored by their hidden states.
Subsequently, the model integrates the outputs of different layers and
harmonizes them on the basis of these attention weights. This innovative
approach equips our model with the ability to capture and exploit richer
cross-modal semantic features, which are essential to command and scene
understanding.

###  4.6 Ablation Study

####  4.6.1 Ablation Study for Core Module

We conducted a rigorous ablation study to determine the effect of core
components in CAVG. As shown in Table 5, specifically, model H, inclusive of
all components, consistently outperforms across the evaluation metric,
illustrating the collective value of the components for optimal performance.
Model A, a reduced version of Model H, replaces the Vision Encoder with the
classical R-CNN, leading to substantial decreases in $IoU_{0.5}$ score by
5.8%, highlighting the importance of visual features in prediction accuracy.
This result also reveals the remarkable superiority of the Vision Encoder
structure over the traditional R-CNN model. The Vision Encoder, with its
single-stage, anchor-free design, streamlines the object detection process. It
locates object centers directly on the feature map and uses regression to
determine object dimensions. This approach not only eliminates the need for
region proposal and anchor adjustment, simplifying the detection process but
also provides a significant increase in detection accuracy.

Model B, which excludes the Context and Emotion Encoder compared to the full
model, shows significant metric reductions, this underscores the essential
role of the additional visual cues in improving prediction accuracy.
Furthermore, Model C, which omits the Text Encoder and Cross-Modal Encoder,
exhibits performance drops of 6.6% in metric, suggesting the importance of
processing instructions and multimodal information individually in this task.
In addition, Model D, which lacks the visual Text Encoder, suffers notable
performance degradation, emphasizing the improved efficacy of the proposed
multi-head cross-modal attention mechanism. Accordingly, Model E and Model F
omit the UNITER embedding process in the Multimodal Decoder and the Emotion
Encoder, respectively, resulting in a decrease in the $IoU_{0.5}$ score as
well. Finally, Model G, which omits the three core components of CAVG, suffers
a noticeable decline in performance compared to Model H. This demonstrates the
significant role that the synergistic integration of various modules plays in
enhancing the accuracy of our model.

Table 5: Component Constituents of Ablation Models.

Components | Ablation Models |
---|---|---
A | B | C | D | E | F | G | H
Vision Encoder | ✘ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
Context Encoder | ✔ | ✘ | ✔ | ✔ | ✔ | ✔ | ✘ | ✔
Text Encoder | ✔ | ✔ | ✘ | ✔ | ✔ | ✔ | ✔ | ✔
Cross-Modal Encoder | ✔ | ✔ | ✘ | ✘ | ✔ | ✔ | ✔ | ✔
UNITER-Embedding | ✔ | ✔ | ✔ | ✔ | ✘ | ✔ | ✘ | ✔
Emotion Encoder | ✔ | ✔ | ✔ | ✔ | ✘ | ✘ | ✘ | ✔
$IoU_{0.5}$ | 71.3 | 66.5 | 68.4 | 70.1 | 71.9 | 73.2 | 64.1 | 74.6

Figure 11: Visualization of Prediction Results for CAVG with and without the
Integration of Context Encoder and the Cross-Modal Attention Mechanism. The
blue box denotes the ground-truth region of interest. The yellow bounding box
represents predictions made by CAVG without the use of a context encoder and
multi-head cross-modal attention mechanisms. The red box shows predictions
from CAVG when these advanced components are incorporated. Figure 12: Raw
Images and their Self-Attention Maps Processed by the Context Encoder. The raw
images are visualized on the left, while the attention maps of these images in
different layers within the encoder are displayed on the right.

####  4.6.2 Case Study for the Context Encoder and Cross-modal Attention

To showcase the effectiveness of the proposed Context Encoder and the cross-
modal attention mechanism and their interpretability. We visualize the
prediction results made by CAVG with and without these components in Fig. 11.
Specifically, we select several challenging examples where CAVG, without the
Context Encoder and cross-modal attention mechanism, made inaccurate
predictions. However, the inclusion of these components markedly improved the
predictive accuracy of CAVG, as evidenced by our thorough evaluation. For a
more detailed understanding, Fig. 12 shows the output of the raw image after
processing by the Context Encoder. This processing emphasizes key scene
semantics in traffic scenarios, such as lane boundaries, road signs, and
pedestrian paths. Moreover, the Context Encoder is able to capture more
nuanced environmental cues, such as variations in lighting or indications of
weather conditions. This extends the visual range of the model beyond the
bounding boxes to a more holistic view of the environment. In addition, the
cross-modal attention mechanism is instrumental in effectively fusing data
from different modalities. Thus, the model goes beyond the visual input and
harmonizes it with the given command.

####  4.6.3 Subtle Value Analysis of the Emotion Encoder

Our ablation study, specifically focusing on the Model F without the Emotion
Encoder, provides a revealing perspective. At first glance, the empirical
improvements attributed to the inclusion of this encoder may seem relatively
modest when measured against the broader testset. However, a deeper dive
reveals the critical role of this component. We argue that the power of the
Emotion Encoder is not captured by broad metrics alone; its true potential
emerges in specific, critical scenarios. For example, in a real-world driving
context, consider a situation in which a visibly anxious passenger commands
the AV to "drive fast"; while the literal interpretation focuses on speed, an
understanding of the underlying emotional urgency might prompt the AV to
prioritize a destination with fewer obstacles. Such human-friendly responses,
attuned to human emotions, could significantly improve passenger comfort.
Although these emotion-laden scenarios might constitute a smaller fraction of
the entire dataset, their impact is disproportionately significant.

Importantly, the inclusion of the Emotion Encoder is strategic and lays the
groundwork for our future efforts. Grounding linguistic commands in a visual
context is only the first step. The next frontier, armed with this
understanding of emotional context, is seamless trajectory planning that not
only adheres to the explicit command but also resonates with the emotional
state of the commander. In essence, by integrating the Emotion Encoder, we
expect to evolve CAVG toward truly empathetic and anticipatory autonomous
driving systems.

###  4.7 Intuition and Interpretability Analysis

CAVG is constructed with several key design motivations. First, a context
encoder captures essential semantic relationships between bounding boxes by
ingesting raw images and textual vectors, thereby enriching cross-modal
understanding. Second, we present an innovative Emotion Encoder that analyzes
the emotional information implicit in commander commands to more deeply
capture human-AV interactions. Furthermore, we introduce a novel cross-modal
attention mechanism to harmonize different types of input data. Finally, the
RSD layer attention mechanism is employed to fine-tune the fusion of multi-
modal vectors and hidden states, enhancing feature interpretability. These
combined elements make CAVG both intuitive and adept at making robust
predictions in complex scenarios.

Figure 13: Illustration of the Emotion Encoder. Demonstrate how the Emotion
Encoder indicates the model to identify a specific vehicle using an emotion
vector embedded in a text vector.

Figure 13 vividly illustrates how our innovative emotion classification
paradigm provides indispensable guidance to CAVG in certain contexts. Given a
commander’s command “Oh, right here! Park behind that white car.”, the
ambiguity inherent in such a directive becomes clear. Without integrating the
commander’s emotional context, the model tends to predict parking in front of
the approaching white vehicle, resulting in a notable deviation from the
ground truth. In contrast, our novel Emotion Encoder delves deep into the
linguistic milieu of the command. Through nuanced linguistic analysis, such as
recognizing the urgency conveyed by exclamatory phrases or specific keywords,
it can identify underlying emotional undertones. In this case, the Emotion
Encoder categorizes the emotion as “urgent”. This classification, when fused
with the text vector produced by the Text Encoder, produces a richer emotion-
sensitive embedding. This fusion communicates to the model the paramount
importance and urgency of the commander’s request. With this emotional
insight, the model can better understand the commander’s immediate needs. This
enhanced understanding allows for the mitigation of potential misjudgments
caused by ambiguous commands while taking into account human emotions for a
more human-centered response.

Figure 14: Illustration of the Cross-Modal Attention Mechanism. Demonstration
of how keywords such as  “just” guide the model in identifying specific
vehicles, using word vectors and regional features. Figure 15: Comparative
Visualization of Model Performance on the Talk2Car Dataset. Ground truth
bounding boxes are depicted in blue, while output bounding boxes of CAVG are
highlighted in red. A natural language command associated with each visual
scenario is also displayed below the image for context.

In addition, in Fig. 14, we demonstrate the crucial role of cross-modal
attention mechanisms in disambiguating natural language commands.
Specifically, the attention to certain keywords in the command can greatly
influence the model’s output. For example, in the depicted scenario, the term
“just” serves as a key keyword. Overlooking this keyword introduces ambiguity
into the system, causing the model to consider multiple vehicles that meet the
given criteria. Our multi-modal attention mechanism addresses this by
correlating word vectors from the command with regional feature vectors in the
image. This allows the model to allocate varying degrees of importance to
different segments of the command. For example, only when the model
incorporates global features like zebra crossings can it understand that a
vehicle has “just” passed the crossing. Consequently, regions that are more
semantically aligned with the keywords in the command receive higher attention
scores. This enhanced attention mechanism captures not just the features
within the bounding boxes but also broader contextual elements, such as lane
markings and zebra crossings, allowing the model to make more informed and
precise decisions.

Figure 16: Comparative Visualization of Model Performance on Challenging
Scenes. The challenging scenes include those with limited visibility,
ambiguous commands, and scenes with multiple agents. Ground truth bounding
boxes are depicted in blue, while output bounding boxes of CAVG are
highlighted in red. A natural language command associated with each visual
scenario is also displayed below the image for context.

###  4.8 Qualitative Results

To demonstrate the reasoning capabilities of our model, we select a set of
challenging driving scenarios for more visual inspection. These selected
scenarios exemplify the diverse and multifaceted situations that a vehicle may
encounter. The visual demonstrations of our model’s performance on these
instances, derived from the Talk2car dataset, are shown in Fig. 15 and Fig.
16. CAVG shows remarkable performance in predicting semantic regions from the
graphical representation, demonstrating its robustness over a range of
environmental conditions. These include challenging conditions such as low-
light night scenes, urban environments with complex object interactions,
ambiguous command scenes, and rainy days where visibility is compromised.

##  5 User Study

To assess user satisfaction with our model and SOTA models in real-world
applications, we conducted a comprehensive user study using questionnaires.
The purpose of this study is to gather user experience and feedback on various
visual grounding models for AVs. By analyzing their responses, we sought to
evaluate these models in detail and identify their strengths and areas for
improvement. This user-centered approach reveals the real-world applicability
and effectiveness of these models and contributes to the development of user-
friendly AV systems.

Recognizing that user satisfaction is closely related to the driving
experience and skills of the participants, our survey included a diverse group
of 174 participants, categorized by gender, age, driving experience, and
educational background. As shown in Fig. 17, the group of participants is
evenly distributed and includes users with varying levels of driving
experience, including those with extensive experience (over ten years) and
those with less experience (one to three years). These participants evaluate
five different visual grounding models (CAVG, CAVG (75%), CAVG (50%), Stacked
VL-BERT, CMSVG, and AttnGrounder) based on three key performance indicators
(response accuracy, inference time, and user experience), and select the most
preferred model from among them. Notably, to ensure fairness, the survey is
anonymous; participants are unaware of the specific categories of the five
models. Details of the survey can be found in the Appendix.

Figure 17: Age and Driving Experience Distribution of Survey Participants.

Figure 18 (a) shows a five-level grading system for the three key user
experience metrics, with scores ranging from 0 to 5. CAVG scores highest in
terms of response accuracy and user experience alignment, while also
maintaining the faster response time. In addition, both the CAVG and CAVG
(75%) provide an impressive user experience and receive relatively high user
ratings compared to other models. Overall, CAVG outperforms the existing
models by 5.2% to 10.6% in terms of comprehensive user ratings. Figure 18 (b)
shows the participants’ preferences for the five models. Among the preferred
models, CAVG receives the most favor from the participants, indicating CAVG’s
practicality and significantly higher user satisfaction compared to other
models. Crucially, feedback from the survey participants indicates that our
future work should focus on further optimizing the structure of the model.
This would involve balancing prediction accuracy with improved inference speed
to respond more promptly to user commands and meet the real-time demands of
visual grounding for autonomous driving, where immediate and accurate response
is paramount for safety and user satisfaction.

Figure 18: Evaluation Results for the User Study. (a) Average scores of the
models in response accuracy, inference time, and user experience. (b)
Quantitative Distribution of favorite models selected by survey participants.

##  6 Conclusion

In this research, we have developed a CAVG framework, meticulously crafted to
enhance visual grounding in AVs. This framework is an integration of five core
encoders—Text, Emotion, Vision, Context, and Cross-Modal—each contributing
uniquely to the processing of multi-modal data, alongside a Multimodal
Decoder. The Text Encoder, utilizing the BERT framework, plays a crucial role
in extracting the linguistic nuances within commander commands. It ensures
that the subtleties of language are accurately interpreted. Complementing
this, our Emotion Encoder, leveraging the capabilities of GPT-4, is pivotal in
understanding and classifying the emotional context of commands. This feature
enables AVs to respond to human commands in a more empathetic and contextually
aware manner. The Vision Encoder is a key component of our model, going beyond
conventional object recognition. It employs frameworks like ViT and BLIP to
provide a richer contextual understanding of traffic scenes. This capability
is crucial for accurately interpreting and responding to grounding commands
with a broader range of visual information. Furthermore, the Cross-Modal
Encoder introduces a multi-head cross-modal attention mechanism. This
mechanism adeptly combines textual and visual information, focusing attention
on the most pertinent vectors. The Multimodal Decoder then employs RSD layer
attention to fuse insights from each layer, leading to informed multimodal
predictions. Tested on the Talk2Car dataset, CAVG demonstrates exceptional
prediction accuracy, surpassing existing state-of-the-art methods. Notably, it
maintains superior performance even with limited training data, ranging from
50% to 75% of the full dataset. Its robustness is further evident under
challenging scenarios such as low-light night scenes, long-text command
scenarios, rainy days, and dense urban streets.

In conclusion, CAVG not only sets a new benchmark in the AV domain but also
significantly enhances the safety and human-centric aspect of the autonomous
driving experience. Future research will focus on expanding CAVG’s
capabilities, including integrating additional modalities like bird’s-eye-view
cues, lidar feedback, and HD maps. We also plan to investigate CAVG’s
integration into a comprehensive AV framework, particularly aligning with
trajectory planning tasks.

## Acknowledgement

This research is supported by the Science and Technology Development Fund of
Macau SAR (File no. 0021/2022/ITP, 0081/2022/A2, SKL-
IoTSC(UM)-2021-2023/ORP/GA08/2022, SKL-IoTSC(UM)-2024-2026/ORP/GA06/2023), and
University of Macau (SRG2023-00037-IOTSC). We extend our gratitude to the
Institute for Advanced Studies in Humanities and Social Sciences (IAS) for
their valuable assistance. For any correspondence regarding this paper, please
contact Dr. Zhenning Li at<EMAIL_ADDRESS>or Dr. Chengzhong Xu at
<EMAIL_ADDRESS>
## References

  * [1] Zhenning Li, Zhiwei Chen, Yunjian Li, and Chengzhong Xu.  Context-aware trajectory prediction for autonomous driving in heterogeneous environments.  Computer-Aided Civil and Infrastructure Engineering, 2023.
  * [2] Guofa Li, Yifan Qiu, Yifan Yang, Zhenning Li, Shen Li, Wenbo Chu, Paul Green, and Shengbo Eben Li.  Lane change strategies for autonomous vehicles: a deep reinforcement learning approach based on transformer.  IEEE Transactions on Intelligent Vehicles, 2022.
  * [3] Thierry Deruyttere, Victor Milewski, and Marie-Francine Moens.  Giving commands to a self-driving car: How to deal with uncertain situations?  Engineering applications of artificial intelligence, 103:104257, 2021.
  * [4] Kareem Othman.  Public acceptance and perception of autonomous vehicles: a comprehensive review.  AI and Ethics, 1(3):355–387, 2021.
  * [5] Jean-François Bonnefon, Azim Shariff, and Iyad Rahwan.  The social dilemma of autonomous vehicles.  Science, 352(6293):1573–1576, 2016.
  * [6] Jiqian Dong, Sikai Chen, Mohammad Miralinaghi, Tiantian Chen, Pei Li, and Samuel Labi.  Why did the ai make that decision? towards an explainable artificial intelligence (xai) for autonomous driving systems.  Transportation research part C: emerging technologies, 156:104358, 2023.
  * [7] Duyu Tang, Bing Qin, and Ting Liu.  Learning semantic representations of users and products for document level sentiment classification.  In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1014–1023, 2015.
  * [8] Vivek Mittal.  Attngrounder: Talking to cars with attention.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 62–73. Springer, 2020.
  * [9] Thierry Deruyttere, Guillem Collell, and Marie-Francine Moens.  Giving commands to a self-driving car: A multimodal reasoner for visual grounding.  arXiv preprint arXiv:2003.08717, 2020.
  * [10] Nivedita Rufus, Unni Krishnan R Nair, K Madhava Krishna, and Vineet Gandhi.  Cosine meets softmax: A tough-to-beat baseline for visual grounding.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 39–50. Springer, 2020.
  * [11] Abhidip Bhattacharyya, Cecilia Mauceri, Martha Palmer, and Christoffer Heckman.  Aligning images and text with semantic role labels for fine-grained cross-modal understanding.  In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4944–4954, 2022.
  * [12] Jianfeng Dong, Zhongzi Long, Xiaofeng Mao, Changting Lin, Yuan He, and Shouling Ji.  Multi-level alignment network for domain adaptive cross-modal retrieval.  Neurocomputing, 440:207–219, 2021.
  * [13] Keyu Wen, Jin Xia, Yuanyuan Huang, Linyang Li, Jiayan Xu, and Jie Shao.  Cookie: Contrastive cross-modal knowledge sharing pre-training for vision-language representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2208–2217, 2021.
  * [14] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang.  Vision-language transformer and query generation for referring segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16321–16330, 2021.
  * [15] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang.  Vlt: Vision-language transformer and query generation for referring segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [16] Jiqian Dong, Sikai Chen, Mohammad Miralinaghi, Tiantian Chen, and Samuel Labi.  Development and testing of an image transformer for explainable autonomous driving systems.  Journal of Intelligent and Connected Vehicles, 5(3):235–249, 2022.
  * [17] Dusan Grujicic, Thierry Deruyttere, Marie-Francine Moens, and Matthew B Blaschko.  Predicting physical world destinations for commands given to self-driving cars.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 715–725, 2022.
  * [18] Hou Pong Chan, Mingxi Guo, and Cheng-Zhong Xu.  Grounding commands for autonomous vehicles via layer fusion with region-specific dynamic layer attention.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12464–12470. IEEE, 2022.
  * [19] Kanishk Jain, Varun Chhangani, Amogh Tiwari, K Madhava Krishna, and Vineet Gandhi.  Ground then navigate: Language-guided navigation in dynamic scenes.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 4113–4120. IEEE, 2023.
  * [20] Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo.  Real-time referring expression comprehension by single-stage grounding network.  arXiv preprint arXiv:1812.03426, 2018.
  * [21] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo.  A fast and accurate one-stage approach to visual grounding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4683–4693, 2019.
  * [22] Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li.  A real-time cross-modality correlation filtering method for referring expression comprehension.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10880–10889, 2020.
  * [23] Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo.  Improving one-stage visual grounding by recursive sub-query construction.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 387–404. Springer, 2020.
  * [24] Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu.  Improving visual grounding with visual-linguistic verification and iterative reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9499–9508, 2022.
  * [25] Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel.  Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1960–1968, 2019.
  * [26] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [27] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik.  Learning two-branch neural networks for image-text matching tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):394–407, 2018.
  * [28] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton Van Den Hengel.  Parallel attention: A unified framework for visual object discovery through dialogs and queries.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4252–4261, 2018.
  * [29] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg.  Mattnet: Modular attention network for referring expression comprehension.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1307–1315, 2018.
  * [30] Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott.  Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language berts.  Transactions of the Association for Computational Linguistics, 9:978–994, 2021.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [33] Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens.  Talk2car: Taking control of your self-driving car.  arXiv preprint arXiv:1909.10838, 2019.
  * [34] Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Yu Liu, Luc Van Gool, Matthew Blaschko, Tinne Tuytelaars, and Marie-Francine Moens.  Commands 4 autonomous vehicles (c4av) workshop summary.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 3–26. Springer, 2020.
  * [35] Nivedita Rufus, Kanishk Jain, Unni Krishnan R Nair, Vineet Gandhi, and K Madhava Krishna.  Grounding linguistic commands to navigable regions.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8593–8600. IEEE, 2021.
  * [36] Wenhao Cheng, Junbo Yin, Wei Li, Ruigang Yang, and Jianbing Shen.  Language-guided 3d object detection in point cloud for autonomous driving.  arXiv preprint arXiv:2305.15765, 2023.
  * [37] Zichao Dong, Weikun Zhang, Xufeng Huang, Hang Ji, Xin Zhan, and Junbo Chen.  Hubo-vlm: Unified vision-language model designed for human robot interaction tasks.  arXiv preprint arXiv:2308.12537, 2023.
  * [38] Zhenning Li, Haicheng Liao, Ruru Tang, Guofa Li, Yunjian Li, and Chengzhong Xu.  Mitigating the impact of outliers in traffic crash analysis: A robust bayesian regression approach with application to tunnel crash data.  Accident Analysis & Prevention, 185:107019, 2023.
  * [39] Zhenning Li, Chengzhong Xu, and Zilin Bian.  A force-driven model for passenger evacuation in bus fires.  Physica A: Statistical Mechanics and its Applications, 589:126591, 2022.
  * [40] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [41] Kisu Yang, Dongyub Lee, Taesun Whang, Seolhwa Lee, and Heuiseok Lim.  Emotionx-ku: Bert-max based contextual emotion classifier.  arXiv preprint arXiv:1906.11565, 2019.
  * [42] Kiana Kheiri and Hamid Karimi.  Sentimentgpt: Exploiting gpt for advanced sentiment analysis and its departure from current machine learning.  arXiv preprint arXiv:2307.10234, 2023.
  * [43] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  arXiv preprint arXiv:2001.07966, 2020.
  * [44] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530, 2019.
  * [45] OpenAI.  Gpt-4 technical report, 2023.
  * [46] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [47] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell.  Natural language object retrieval.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4555–4564, 2016.
  * [48] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [49] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  Advances in Neural Information Processing Systems, 34:17864–17875, 2021.
  * [50] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [51] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021.
  * [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [53] Siyu Hao, Der-Horng Lee, and De Zhao.  Sequence to sequence learning with attention mechanism for short-term passenger flow prediction in large-scale metro system.  Transportation Research Part C: Emerging Technologies, 107:287–300, 2019.
  * [54] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In European conference on computer vision, pages 104–120. Springer, 2020.
  * [55] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [56] Mark Everingham and John Winn.  The pascal visual object classes challenge 2012 (voc2012) development kit.  Pattern Anal. Stat. Model. Comput. Learn., Tech. Rep, 2007(1-45):5, 2012.
  * [57] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88:303–338, 2010.
  * [58] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [59] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [60] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [61] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [62] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [63] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [64] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Hierarchical question-image co-attention for visual question answering.  Advances in neural information processing systems, 29, 2016.
  * [65] Simon Vandenhende, Thierry Deruyttere, and Dusan Grujicic.  A baseline for the commands for autonomous vehicles challenge.  arXiv preprint arXiv:2004.13822, 2020.
  * [66] Drew A Hudson and Christopher D Manning.  Compositional attention networks for machine reasoning.  arXiv preprint arXiv:1803.03067, 2018.
  * [67] Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan.  Visual grounding via accumulated attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7746–7755, 2018.
  * [68] Hang Dai, Shujie Luo, Yong Ding, and Ling Shao.  Commands for autonomous vehicles by progressively stacking visual-linguistic representations.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 27–32. Springer, 2020.
  * [69] Shujie Luo, Hang Dai, Ling Shao, and Yong Ding.  C4av: learning cross-modal representations from transformers.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 33–38. Springer, 2020.
  * [70] Jie Ou and Xinying Zhang.  Attention enhanced single stage multimodal reasoner.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 51–61. Springer, 2020.
  * [71] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.