# Instruction-Guided Visual Masking

Jinliang Zheng111Equal contribution 1,2, Jianxiong Li111Equal contribution 1,
Sijie Cheng1, Yinan Zheng1,
Jiaming Li1, Jihao Liu3,2, Yu Liu2, Jingjing Liu222Corresponding author 1,
Xianyuan Zhan222Corresponding author 1,4
1 AIR, Tsinghua University, 2 Sensetime Research
3 MMLab, CUHK, 4 Shanghai AI Lab
{zhengjl23<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Instruction following is crucial in contemporary LLM. However, when extended
to multimodal setting, it often suffers from misalignment between specific
textual instruction and targeted local region of an image. To achieve more
accurate and nuanced multimodal instruction following, we introduce
Instruction-guided Visual Masking (IVM), a new versatile visual grounding
model that is compatible with diverse multimodal models, such as LMM and robot
model. By constructing visual masks for instruction-irrelevant regions, IVM-
enhanced multimodal models can effectively focus on task-relevant image
regions to better align with complex instructions. Specifically, we design a
visual masking data generation pipeline and create an IVM-Mix-1M dataset with
1 million image-instruction pairs. We further introduce a new learning
technique, Discriminator Weighted Supervised Learning (DWSL) for preferential
IVM training that prioritizes high-quality data samples. Experimental results
on generic multimodal tasks such as VQA and embodied robotic control
demonstrate the versatility of IVM, which as a plug-and-play tool,
significantly boosts the performance of diverse multimodal models, yielding
new state-of-the-art results across challenging multimodal benchmarks. Code is
available at https://github.com/2toinf/IVM.

##  1 Introduction

Multimodal instruction following is a fundamental multimodal task, powering a
wide-range of applications such as visual question answering (VQA) [18],
visual captioning [1, 41], and embodied robotic control [14]. To effectively
solve this task, one critical capability required is nuanced image-language
grounding, which current multimodal models grow implicitly and slowly through
data-intensive end-to-end training without explicit grounding supervisions.
Two challenges emerge in this indirect learning of image-instruction
alignment: $1)$ How to accurately localize targeted image regions that
corresponds to a specific textual instruction, as illustrated in Figure 2.
$2)$ How to generalize to diverse visual representations (e.g., same object
with different colors, compositions, or backgrounds) that reflect similar
textual instruction (e.g., Q3 in Figure 2). Lacking an effective and direct
solution to these challenges, the most advanced Large Multimodal Models (LMMs)
[1, 6, 41, 14] still suffer from hallucinations even when trained with high-
quality data in the magnitude of billions [34].

Figure 1: The most advanced LMMs (e.g. GPT4-V) still fail on complex
instruction following tasks. With IVM assistance to simplify visual inputs,
existing LMMs can gain significant improvement.

Figure 2: Comparison between IVM and Reasoning Segmentation (RS) [31].
Traditional methods such as semantic segmentation [68] and referring
expression comprehension [64] are limited to fixed categories or fixed
instruction formats, thus inapplicable to complex instruction following tasks.
RS has reasoning ability, but only allows single object localization. IVM,
instead, is universally applicable to any instruction.

We introduce Instruction-guided Visual Masking (IVM), a versatile plug-and-
play model designed to enhance multimodal instruction following via nuanced
surgical visual grounding. To eliminate the distraction of instruction-
irrelevant visual regions, IVM automatically masks out these regions to
sharpen the focus of instruction following, and meticulously crops visual
input to tailor for a specific instruction and enforce multimodal models to
zoom in on task-related visual content. Existing visual grounding methods are
limited either to predefined object categories, which cannot cover diverse
instruction-related visual content; or they subscribe to a fixed instruction
format, which restricts the expressiveness of instructions. As shown in Figure
2, such simplistic grounding techniques often fail to comprehend complex
instruction-following tasks.

Learning an IVM model requires pixel-level, fine-grained, instruction-guided
mask annotations that provide explicit grounding supervisions. To create such
a dataset, we build a LLM-empowered Mixture of Expert pipeline with SOTA
visual grounding models [52, 50, 31, 20] to efficiently create abundant
reliable labels. To compensate the noises in auto-generated labels, we further
manually label a smaller dataset with clean annotations, and integrate the two
into an IVM-Mix-1M dataset that contains 1 million image-instruction pairs.

To reduce demand on costly human labels and ensure optimized utility of
machine-generated labels, we employ a Discriminator-Weighted Supervised
Learning (DWSL) framework for IVM training, inspired by recent advances in
offline imitation learning [60]. Specifically, we introduce a discriminator to
assign weights to masks, where high values are assigned to high-quality
annotations and vice versa. Thus, these weights generated by the discriminator
can naturally act as a weighting function for the IVM training objective,
allowing for a preferential training process that prioritizes learning from
reliable samples and discards misleading ones.

Extensive experiments demonstrate great versatility of the IVM model when
integrated into existing multimodal chatbots (commercial and open-sourced)
without fine-tuning. Our IVM-enhanced LMMs gain significant performance
improvement across new challenging benchmarks such as V*Bench [58], EgoThink
[10] and POPE [34], achieving new state of the art. IVM model also proves
valuable in vision-language robotic manipulation tasks, where data collection
is notoriously challenging and generalization is a major concern [35]. With
the integration of IVM, our enhanced robot model exhibits boosted performance
and better generalization capabilities.

Our contributions are summarized as follows: $1)$ We propose Instruction-
guided Visual Masking (IVM), a novel approach that serves as a versatile plug-
and-play module to enhance multimodal models through visual grounding. $2)$ We
introduce the IVM-Mix-1M dataset and propose an LLM-empowered Mixture of
Expert pipeline to create visual grounding labels. $3)$ We present the DWSL
algorithm for IVM training that automatically prioritizes high-quality
training samples.

##  2 Related Work

Large Multimodal Models. LLaVA [41] first demonstrates promising capabilities
in following complex instructions. Subsequent works such as LLaVA-1.5 [38],
MiniGPT4 [69] Qwen-VL [6] and CogVLM [57], further enhance LMMs via refined
model design and enriching the quality of training data, achieving state-of-
the-art performance on diverse downstream tasks including visual grounding
[36], visual reasoning [54], visual question and answering [18]. Moreover, by
integrating the robotics action modality, LMMs perform versatile planning and
manipulation in instruction-driven robotics tasks. Notable studies in this
line of inquiry include PaLM-E [14], the series of RT models [8, 9, 55], and
text-guided video planning diffusion models [15, 62, 7]. Despite the success,
LMMs still struggle with complex visual grounding challenges, often misreading
instruction-irrelevant visual contents (Figure 2). To address this,
researchers have tried to adapt existing visual modules to higher-resolution
images to obtain better perception [40], but with limited improvement.

Visual Grounding Tasks. Visual grounding requires precisely localizing image
regions corresponding to a referring expression, among which the RefCOCO
series [64] is the most well-known benchmark, and numerous public visual
grounding data are available [36, 63, 19]. Recently, LMMs incorporate these
visual grounding data via visual-instruction tuning [41, 38, 69], establishing
new SOTA in this area [50]. To further broaden the reasoning ability of visual
grounding, LISA [31] introduces a new task, reasoning segmentation, which
demands higher capabilities in instruction comprehension. However, visual
grounding is still limited to align simple instruction with specific objects,
which cannot adapt to more complex instruction following tasks (e.g. Figure
2).

Visual Grounding Augmented LMMs. Recently, a series of visual grounding
methods emerged to enhance the performance of LMMs in complex visual scenes.
V* [58] employs a heuristic search strategy to search, locate, and crop image
areas relevant to instructions through a multi-step iterative process.
VisualCot [50] is trained end-to-end with a customized dataset to achieve
target localization capabilities. These two methods allow LMMs to dynamically
focus on visual inputs until the correct answer is derived. However, these
complex inference pipelines lead to substantial computational overhead, and
their heuristic designs further hinder the extension beyond VQA to other
multimodal instruction following tasks such as robotic control.

Besides these explicit strategies incorporating additional visual grounding
modules, other studies pursue refining data or introducing extra training
targets to enhance the grounding capabilities of LMMs implicitly. ViGor [61]
proposes a fine-grained reward modeling to enhance visual grounding of LMMs,
and SynGround [23] introduces a pragmatic framework for image-text-box
synthesis tailored for visual grounding. These methods, however, are primarily
focused on the visual grounding task itself, overlooking its influence on
downstream multimodal instruction following tasks.

Distinct from previous efforts, this paper introduces a generic visual
grounding model that is adaptable to any multimodal instruction following
tasks, and provides a systematic investigation into the advantages of
integrating an additional visual grounding model into downstream applications.

##  3 Instruction-Guided Visual Masking

To help multimodal models focus on instruction-sensitive image regions without
distractions from irrelevant visual elements, we introduce Instruction-guided
Visual Masking (IVM), a versatile plug-and-play model that enhances multimodal
instruction following via surgical targeted visual grounding.

###  3.1 Problem Definition

Figure 3: Instruction-guided Visual Masking.

IVM aims to produce a heatmap $\mathbf{H}$, given an image $\mathbf{x}_{\rm
img}$ and a textual instruction $\mathbf{x}_{\rm txt}$. The heatmap
$\mathbf{H}$ identifies the critical image region to follow the instructions,
as illustrated in Figure 3, allowing multimodal models to easily zoom in on
targeted image regions while ignoring neighboring areas.

This formulation evokes the problem definition of Reasoning Segmentation (RS)
[31]. There are two main differences: $1)$ IVM addresses a more challenging
problem. RS tries to target single objects from simple instructions, e.g.,
"what is.., where is.., who is…", while IVM aims to include all instruction-
related visual regions within the image given any instruction, which demands
advanced and nuanced image-language grounding ability (as illustrated in
Figure 2). $2)$ RS has clear ground truths but IVM does not. The instructions
in RS primarily correspond to simple and semantic-meaningful objects that are
straightforward for human annotations. IVM, however, deals with broader and
more ambiguous instruction-related regions (e.g., the left bank regions in
Figure 3), making the training and annotating much more challenging.

###  3.2 Data Preparation

Figure 4: LLM-empowered Mixture-of-Expert pipeline for auto-annotation. (1)
For labeled VG data, we utilize an LLM to generate complex instruction
annotations. (2) For unlabeled VIF or robot data, we first use an LLM to
simplify the instruction and then leverage a mixture of VG models to generate
candidate labels.

To train an IVM model, the first main challenge is the scarcity of training
data. Most existing Visual Grounding (VG) datasets [64, 31] typically feature
simple instructions focused primarily on prominent objects within images,
lacking both diversity and complexity required for IVM. To tackle this, we
compiled one million data from various sources, including labeled visual
grounding, unlabeled multimodal instruction following, and robotics data. As
outlined in Section 3.1, scaling human annotations is challenging due to the
high complexity of such data. Therefore, we introduce an LLM-empowered Mixture
of Expert pipeline that integrates SOTA visual grounding models to efficiently
generate reliable annotations. We further manually annotate a smaller dataset
to compensate inaccuracies in auto-generated labels. The resulted combined
dataset, IVM-Mix-1M, comprises one million data samples ready for IVM
training, which will be released for future study upon acceptance.

LLM-empowered Mixture of Expert Annotation Pipeline. Leveraging the power of
LLM, this pipeline can efficiently generate high-quality annotation, which
consists of two components (Figure 4): $1)$ Labeled visual grounding data. We
collect 250K labeled VG data from multiple sources including VG caption [36],
Flickr30K [63], VSR [3], OpenImage [30], and RefCoCo [64, 37], which provide
bounding boxes with simple instructions for each image. To increase the
diversity and complexity of instructions, we utilize GPT-4 [1], known for its
robust language understanding and generation capabilities, to create diverse
instruction-answer pairs based on existing language instructions. $2)$
Unlabeled Visual-Instruction-Following (VIF) and robotics data. We sample a
700K subset from LLaVA-Instruction-tuning [41] for VQA-type data, and a 50K
subset from OpenX [55] for robotics data. Given that these data lack grounded
labels but contain complex instructions, we use GPT-4 to simplify the language
instructions by prompting it to infer the names of targeted objects necessary
for following the instructions. These simplified instructions then guide
existing VG models to generate candidate labels. To ensure the quality of
these labels and compensate for the ambiguous nature of the IVM task, we
integrate proposals from several VG experts, such as Grounding-Sam [49], LISA
[31], AlphaClip [52], and OwlViT [20], via an ensemble approach.

Manual Annotation. Despite integrating the most advanced models, the auto-
generation design still faces challenges that can lead to data inaccuracies.
First, employing LLM to simplify or complicate language annotations without
considering image content can introduce uncontrollable biases. Second, as the
task exceeds the capabilities of existing models, it becomes impossible to
totally exclude low-quality annotations that contain irrelevant visuals or
mistakenly filter out critical contents. Thus, to enhance the overall quality
of the dataset, we further manually annotate a 10K subset of the constructed
dataset to inject human expert knowledge.

Figure 5: Data analysis on the IVM-Mix-1M dataset: data quantity v.s
percentage of instruction-related areas.

Data Analysis. Here, we provide quantitative analysis on the IVM-Mix-1M
dataset. Figure 5 depicts the data quantities w.r.t the percentage of
annotated instruction-related image area. Here, each ratio range is further
categorized by different data sources, where manually annotations are treated
as a separate category (Human), while all others are machine-generated. Our
analysis reveals that the instruction-related image regions only occupy a
small fraction of the total image area (e.g. most data have less than 40%
instruction-relevant image regions), indicating that most visual contents may
cause distraction and corroborating the necessity of visual masking for
instruction following tasks.

###  3.3 Discriminator-Weighted Supervised Learning Framework

The challenge now is to train the IVM model with a small high-quality human-
annotated dataset ($\mathcal{D}_{e}$) as well as a large but mixed-quality
auto-generated dataset ($\mathcal{D}_{o}$). Training naively on the combined
dataset may yield suboptimal results due to inaccuracies in auto-generated
labels, while solely using limited human-annotated data is insufficient.
Inspired by recent advances in imitation learning using mixed-quality data
[60, 65], we employ a Discriminator-Weighted Supervised Learning (DWSL)
framework to effectively leverage the strengths of both auto- and human-
annotated data.

Figure 6: IVM model architecture. A frozen vision backbone and a LORA-tuned
LMM are utilized to extract dense image features and multimodal
representations, respectively. These features are then fed into a generator
for dense prediction and a discriminator for generating training weights. See
Appendix C.1 for more details.

Discriminator Training. Specifically, we introduce a discriminator $d$
optimized by Eq. (1) to assign high weights to high-quality annotations and
vice versa:

$\displaystyle\min_{d}\mathbb{E}_{(\mathbf{x}_{\rm img},\mathbf{x}_{\rm
txt},\mathbf{H})\sim\mathcal{D}_{e}}$ $\displaystyle\left[-\log
d(\mathbf{x}_{\rm img},\mathbf{x}_{\rm
txt},\mathbf{H})\right]+\mathbb{E}_{(\mathbf{x}_{\rm img},\mathbf{x}_{\rm
txt},\mathbf{H})\sim\mathcal{D}_{o}}\left[-\log(1-d(\mathbf{x}_{\rm
img},\mathbf{x}_{\rm txt},\mathbf{H}))\right],$ (1)

where $(\mathbf{x}_{\rm img},\mathbf{x}_{\rm txt},\mathbf{H})$ are image-
instruction-heatmap pairs sampled from $\mathcal{D}_{o}$ and $\mathcal{D}_{e}$
datasets. Eq. (1) is similar to the one in GAN [17], but the "fake" data in
[17] is replaced by machine-generated data from $\mathcal{D}_{o}$. After
training with Eq. (1), the discriminator $d$ assigns high weights to high-
quality human-annotated data from $\mathcal{D}_{e}$ and relatively high values
to similarly high-quality data from $\mathcal{D}_{o}$ that aligns with human
preferences, acting as a judge for annotation quality.

Discriminator-weighted IVM Training. Then, we apply the trained discriminator
as a weighting function for the IVM training objective:

$\displaystyle\min_{\theta}$ $\displaystyle\mathbb{E}_{(\mathbf{x}_{\rm
img},\mathbf{x}_{\rm
txt},\mathbf{H})\sim\mathcal{D}_{o}\cup\mathcal{D}_{e}}\left[f\left(d(\mathbf{x}_{\rm
img},\mathbf{x}_{\rm txt},\mathbf{H})\right)\mathcal{L}^{\rm
IVM}_{\theta}(\mathbf{x}_{\rm img},\mathbf{x}_{\rm txt},\mathbf{H})\right],$
(2) $\displaystyle\mathcal{L}^{\rm IVM}_{\theta}(\mathbf{x}_{\rm
img},\mathbf{x}_{\rm txt},\mathbf{H})=\lambda_{\rm
bce}\mathbf{BCE}(\hat{\mathbf{H}}_{\theta},\mathbf{H})+\lambda_{\rm
dice}\mathbf{DICE}(\hat{\mathbf{H}}_{\theta},\mathbf{H}),$ (3)

where $\lambda_{\rm bce}$ and $\lambda_{\rm dice}$ are set to 1.0 and 1.0 to
balance the binary cross-entropy loss ($\mathbf{BCE}$) and the DICE loss for
segmentation ($\mathbf{DICE}$) [28], respectively. $f(x)\geq 0$ can be any
non-negative, non-decreasing function. For simplicity, we set
$f(x):=\min(\max(0.1,x),1)$. This allows the weighting function $f(d(\cdot))$
in Eq. (2) to dynamically prioritize training with high-quality data
determined by the discriminator $d$. This approach maximizes the usage of
reliable annotations in $\mathcal{D}_{o}$ to compensate for the small
$\mathcal{D}_{e}$, while minimizing the impact of low-quality data in
$\mathcal{D}_{o}$, thus optimizing performance.

###  3.4 Model Architecture

The overall model framework is illustrated in Figure 6. Due to its complexity,
IVM requires both reasoning and precise localization of the target object,
closely paralleling reasoning segmentation [31]. Consequently, for the heatmap
generator, we adopt a model design similar to that of LISA [31]. Specifically,
we first extract dense image features using an isolated vision backbone and
multimodal representation from an LMM, which processes image-instruction
pairs. These two types of features are then fed into a lightweight generator
that integrates them to produce a dense prediction.

For the discriminator, we deploy a lightweight discriminator that encodes the
segmentation map using a two-layer convolution network. This discriminator
interacts with the outputs of the LMMs through multiple cross-attention
operators and finally outputs a quality score for each sample.

Trainable Parameters. To enhance training efficiency, we freeze the pre-
trained large foundation models and perform LORA finetuning [25]. The vision
backbone, inherited from Segment Anything Model [29], is completely frozen,
while the lightweight generator and discriminator are fully finetuned.
Notably, we utilize a shared LMM for both the generator and discriminator
branches but employing separate LORA parameters to avoid interference between
the two tasks.

##  4 Experiments

In this work, we employ LLaVA-7B [42] as the LMM and SAM-H [29] as the vision
backbone for our IVM model (Figure 6), which is trained on the IVM-Mix-1M
dataset using the proposed DWSL algorithm. More details on the architecture
and training can be found in Appendix C. We conduct extensive experiments to
assess the effectiveness of the IVM model. Specifically, we utilize the
heatmap generated by the IVM for image post-processing. These processed images
can then be seamlessly fed into downstream multimodal models for diverse
tasks. Unless otherwise specified, we use the image post-processing method of
overlaying and cropping to discard instruction-irrelevant image content. A
detailed discussion on post-processing methods is presented in Section 10. We
also provide more evaluation results and analysis in Appendix E.

###  4.1 Main Results

Table 1: V* bench results.

LMMs | Attribute(%) | Spatial(%) | Overall(%)
---|---|---|---
Open-Sourced LMMs
BLIP2 [33] | 27.0 | 53.9 | 37.7
MiniGPT-4 [69] | 30.4 | 50.0 | 38.2
InstructBLIP [12] | 25.2 | 47.4 | 34.0
Otter [32] | 27.0 | 56.6 | 38.7
LLaVA-1.5 [38] | 43.5 | 56.6 | 48.7
Commercial Chatbots
Bard [45] | 31.3 | 46.1 | 37.2
Gemini-Pro [53] | 40.9 | 59.2 | 48.2
GPT4-V [1] | 51.3 | 60.5 | 55.0
Specific Visual Search Models
SEAL [58] | 74.8 (+23.5) | 76.3 (+15.8) | 75.4 (+20.4)
IVM-Enhanced GPT4-V | 87.0 (+35.7) | 72.4 (+11.9) | 81.2 (+26.2)

Figure 7: IVM can handle various instructions, ranging from retaining entire
images for captioning (row 1) to localizing unique objects (row 2 and 3).

Table 2: Results on other multimodal benchmarks. MME* denotes the aggregate of
scores from -p and -c.

LMMs | #Param | EgoThink | POPE | MME* | GQA | SQA | VQAv2
---|---|---|---|---|---|---|---
InstructBLIP [12] | 13B | - | 78.9 | 1212.8 | 49.5 | 60.5 | -
Qwen-7B [6] | 7B | - | - | - | 58.3 | 67.1 | 78.8
SEAL-7B [58] | 7B | - | 82.4 | 1129 | - | - | -
LLaVA-7B [38] | 7B | 51.1 | 85.9 | 1748 | 62.0 | 70.2 | 78.5
LLaVA-13B [38] | 13B | 55.2 | 85.9 | 1834 | 67.1 | 71.6 | 80.0
LISA [31]-Enhanced LLaVA-7B | 20B | 47.9 (-3.2) | 80.0 (-5.9) | 1560 (-188) | 56.6 (-5.4) | 69.3 (-0.9) | 78.2 (-0.3)
IVM-Enhanced LLaVA-7B | 14B | 54.5 (+3.4) | 87.2 (+1.3) | 1806 (+58) | 62.2 (+0.2) | 70.2 (-) | 79.0 (+0.5)

Integration with Commercial Chatbot. We use GPT4-V [1] as the base model.
Considering the superior perception and reasoning capability of GPT4-V, we
evaluate IVM-enhanced GPT4-V on V*bench [58], a recently proposed challenging
VQA-type benchmark characterized by images with abundant redundancies. Results
are presented in Table 4.1. The accuracy of the vanilla GPT4-V is mediocre
(55.0%). Our IVM model, however, can significantly improve the performance
(+26.2%) and establish a new state of the art on this benchmark, even
surpassing the task-specialized SEAL [58] that requires a complex heuristic
visual search pipeline.

Integration with Open-sourced LMMs. To demonstrate the versatility of our IVM
model, we further integrate it into an open-sourced LMM, LLaVA-7B [38]. We
conduct extensive experiments across various benchmarks, including EgoThink
[10], POPE [34], MME [16], GQA [27], SQA [44], and VQAv2 [18]. As shown in
Table 2, our IVM-enhanced LLava-7B gains consistent performance improvements,
achieving comparable performance to (even surpassing) LLaVA-13B on EgoThink,
POPE and MME. Although IVM-enhanced LLaVA-7B and LLaVA-13B [38] have roughly
the same number of parameters, the latter integrates more powerful pretrained
foundation models. In contrast, our IVM model allows the 7B model to
outperform the 13B model by merely simplifying visual input, further
validating the power of visual masking.

Meanwhile, IVM-enhanced LLaVA-7B does not show significant gains on GQA, SQA
and VQAv2, which is expected, as these benchmarks do not heavily rely on
grounding capabilities: VQAv2 and GQA contain relatively simple visual input
where most regions of the images are instruction-relevant, while SQA primarily
focuses on assessing model reasoning capability.

Comparison with Reasoning Segmentation Model. We also compare against LISA
[31], which is most analogous to IVM. We provide carefully tailored prompts
like "what should we focus on the image to follow the given instruction? Give
me the seg" to extend LISA into visual masking task. However, even with larger
13B model and extensive tuning of input prompt, masks generated by LISA
consistently result in severe performance degradation on all tasks.

Evaluation on Real Robotic Control. We also plug the IVM model into robot
control tasks to help robot model improve generalization. Specifically, we
evaluate a language-conditioned behavior cloning (LCBC) robot agent trained
with or without IVM masked images. Figure 8 clearly demonstrates that without
IVM assistance, the LCBC robot agent suffers from severe performance drop when
noticeable distractions are applied. With IVM assistance, however, the agent
consistently pays close attention to correct instruction-related image
regions, performing robustly against diverse distractions such as human
disturbances and numerous task-irrelevant objects of various colors and
shapes. This demonstrates promising potentials of using IVM to enhance
embodied agents to follow complex instructions in unseen scenes with plenty
distractions.

Figure 8: Real robot results with or without IVM assistance. IVM greatly helps
LCBC agent to overcome major distractions, enjoying better robustness and
generalization. See Appendix C.4 for experiment setups.

###  4.2 Ablation

We ablate the key components of IVM and report overall accuracy improvement(%)
of IVM-Enhanced GPT4-V, evaluated on V* bench [58] due to its high demand on
precise visual grounding abilities.

Figure 9: Ablations on training data and the proposed DWSL framework.

Figure 10: Different mask deployment methods.

Table 3: Ablations on different mask deployment methods on the V*bench.

| Overlay | Blur | Gray-scale
---|---|---|---
w/ crop | +26.2 | +24.4 | +22.1
w/o | +19.1 | +17.2 | +10.2

Training Data. We investigate the impact of IVM-Mix data characteristics on
IVM performance from two key perspectives: $1)$ Large machine-annotated data
volume clearly enhances IVM model performance, as illustrated by the
progressive improvement in Figure 9 (a) with increased machine-annotated data
volume (red, blue and yellow line). This demonstrates the effectiveness of our
proposed LLM-empowered Mixture-of-Expert pipeline in generating reliable data
for IVM training. $2)$ Figure 9 (a) also reveals that incorporating human
annotations significantly boosts training efficiency (red and blue v.s yellow
line), highlighting the critical role of introducing human preferences in IVM-
Mix-1M dataset, despite its relatively small volume compared to machine-
annotated data (only 1:100).

DWSL Framework. We also explore the efficacy of the DWSL framework in Figure 9
(a) by comparing IVM training using: $1)$ DWSL (red line), $2)$ traditional
Supervised Learning (SL) without DWSL (blue line), and $3)$ SL on limited
human data (gray line). The results demonstrate that DWSL effectively
leverages both human- and auto-annotated data, particularly as the volume of
machine-annotated data increases, enjoying higher asymptotic performances.
This is expected as machine-annotated data often contain inaccuracies and
training naively using all these data can lead to suboptimal results.
Meanwhile, the limited human data alone cannot provide satisfactory outcomes.
DWSL, however, addresses these challenges by dynamically prioritizing good
samples and discarding misleading ones, resulting in stable and improved
results. This is further illustrated in Figure 9 (b) which visualizes the
outputs of the discriminator for each sample, where the discriminator can
correctly retain good samples (e.g. Human) and filter out low-quality data
with lower weights.

Mask Deployment Strategy. We investigate the impact of mask deployment
strategy on downstream applications. While more complex solutions such as
visual search algorithms [58] can be employed, our investigation focuses
solely on simpler approaches to understand the intrinsic capabilities of IVM
model. Specifically, we examine four basic masking methods: overlay, blur,
grayscale, and cropping, as illustrated in Figure 10. In particular, for the
crop method, we find the smallest area that retains all the activated (>0)
values in the heatmap and crop it. Table 10 demonstrates that IVM maintains
robustness across all simple post-processing methods, where overlay+crop
enjoys the most performance enhancement and thus is used as our default mask
deployment method.

##  5 Conclusion

In this paper, we introduce Instruction-guided Visual Masking (IVM), a generic
and powerful visual grounding method that enhances broad multimodal
instruction following tasks in a plug-and-play way. By masking out all
instruction-irrelevant image regions, IVM effectively injects superior visual
grounding ability to downstream LMMs non-intrusively, significantly boosting
both commercial and open-sourced LMMs and achieving state-of-the-art results
across numerous challenging multimodal benchmarks. Real robot experiments
further demonstrate the versatility of IVM, showcasing the potential to deploy
IVM to embodied robotic tasks where failures caused by distractions are long-
standing challenges. For further improvement, one promising direction is to
finetune LMMs using IVM-generated heatmap as an additional input channel to
reduce suboptimal heuristics caused by mask deployment methods. However, due
to resource limitation, we leave this for future work and will release our IVM
checkpoint as well as the IVM-Mix-1M dataset to help the community further
explore relevant directions. More discussion on limitations and future
directions can be found in Appendix A.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * Ajay et al. [2022] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal.  Is conditional generative modeling all you need for decision-making?  arXiv preprint arXiv:2211.15657, 2022.
  * Alayrac et al. [2019] Jean-Baptiste Alayrac, Joao Carreira, and Andrew Zisserman.  The visual centrifuge: Model-free layered video representations.  In CVPR, 2019.
  * Aminabadi et al. [2022] Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He.  Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022.
  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862, 2022.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  arXiv preprint arXiv:2308.12966, 2023.
  * Black et al. [2024] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine.  Zero-shot robotic manipulation with pre-trained image-editing diffusion models.  In The Twelfth International Conference on Learning Representations, 2024.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  arXiv preprint arXiv:2212.06817, 2022.
  * Brohan et al. [2023] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  arXiv preprint arXiv:2307.15818, 2023.
  * Cheng et al. [2024] Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu.  Egothink: Evaluating first-person perspective thinking capability of vision-language models, 2024.
  * Chi et al. [2023] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song.  Diffusion policy: Visuomotor policy learning via action diffusion.  arXiv preprint arXiv:2303.04137, 2023.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  ArXiv, abs/2305.06500, 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * Du et al. [2023] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel.  Learning universal policies via text-guided video generation.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * Fu et al. [2023] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  arXiv preprint arXiv:2306.13394, 2023.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollár, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation, 2019.
  * Gupta et al. [2022] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Ow-detr: Open-world detection transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9235–9244, 2022.
  * Hansen-Estruch et al. [2023] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine.  Idql: Implicit q-learning as an actor-critic method with diffusion policies.  arXiv preprint arXiv:2304.10573, 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * He et al. [2024] Ruozhen He, Paola Cascante-Bonilla, Ziyan Yang, Alexander C. Berg, and Vicente Ordonez.  Learning from models and data for visual grounding, 2024.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * Hu et al. [2024] Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang.  Query-policy misalignment in preference-based reinforcement learning.  In The Twelfth International Conference on Learning Representations, 2024.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In CVPR, 2019.
  * Jadon [2020] Shruti Jadon.  A survey of loss functions for semantic segmentation.  In 2020 IEEE conference on computational intelligence in bioinformatics and computational biology (CIBCB), pages 1–7. IEEE, 2020.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * Kuznetsova et al. [2020] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  International Journal of Computer Vision, 128(7):1956–1981, March 2020.
  * Lai et al. [2024] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia.  Lisa: Reasoning segmentation via large language model, 2024.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726, 2023.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * Li et al. [2023c] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  arXiv preprint arXiv:2305.10355, 2023.
  * Li et al. [2024] Jianxiong Li, Jinliang Zheng, Yinan Zheng, Liyuan Mao, Xiao Hu, Sijie Cheng, Haoyi Niu, Jihao Liu, Yu Liu, Jingjing Liu, et al.  Decisionnce: Embodied multimodal representations via implicit preference learning.  arXiv preprint arXiv:2402.18137, 2024.
  * Liang et al. [2019] Yuanzhi Liang, Yalong Bai, Wei Zhang, Xueming Qian, Li Zhu, and Tao Mei.  Vrr-vg: Refocusing visually-relevant relationships, 2019.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.
  * Liu et al. [2023b] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
  * Liu et al. [2024a] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
  * Liu et al. [2024b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.
  * Liu et al. [2024c] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2017.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022.
  * Manyika and Hsiao [2023] James Manyika and Sissie Hsiao.  An overview of bard: an early experiment with generative ai.  AI. Google Static Documents, 2, 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in neural information processing systems, 35:27730–27744, 2022.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of machine learning research, 21(140):1–67, 2020.
  * Ren et al. [2024] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang.  Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
  * Shao et al. [2024] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li.  Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models, 2024.
  * Song et al. [2021] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * Sun et al. [2023] Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang.  Alpha-clip: A clip model focusing on wherever you want, 2023.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * Thrush et al. [2022] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality, 2022.
  * Vuong et al. [2023] Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, Chelsea Finn, Max Du, Moo Jin Kim, Alexander Khazatsky, Jonathan Heewon Yang, Tony Z. Zhao, Ken Goldberg, Ryan Hoque, Lawrence Yunliang Chen, Simeon Adebola, Gaurav S. Sukhatme, Gautam Salhotra, Shivin Dass, Lerrel Pinto, Zichen Jeff Cui, Siddhant Haldar, Anant Rai, Nur Muhammad Mahi Shafiullah, Yuke Zhu, Yifeng Zhu, Soroush Nasiriany, Shuran Song, Cheng Chi, Chuer Pan, Wolfram Burgard, Oier Mees, Chenguang Huang, Deepak Pathak, Shikhar Bahl, Russell Mendonca, Gaoyue Zhou, Mohan Kumar Srirama, Sudeep Dasari, Cewu Lu, Hao-Shu Fang, Hongjie Fang, Henrik I Christensen, Masayoshi Tomizuka, Wei Zhan, Mingyu Ding, Chenfeng Xu, Xinghao Zhu, Ran Tian, Youngwoon Lee, Dorsa Sadigh, Yuchen Cui, Suneel Belkhale, Priya Sundaresan, Trevor Darrell, Jitendra Malik, Ilija Radosavovic, Jeannette Bohg, Krishnan Srinivasan, Xiaolong Wang, Nicklas Hansen, Yueh-Hua Wu, Ge Yan, Hao Su, Jiayuan Gu, Xuanlin Li, Niko Suenderhauf, Krishan Rana, Ben Burgess-Limerick, Federico Ceola, Kento Kawaharazuka, Naoaki Kanazawa, Tatsuya Matsushima, Yutaka Matsuo, Yusuke Iwasawa, Hiroki Furuta, Jihoon Oh, Tatsuya Harada, Takayuki Osa, Yujin Tang, Oliver Kroemer, Mohit Sharma, Kevin Lee Zhang, Beomjoon Kim, Yoonyoung Cho, Junhyek Han, Jaehyung Kim, Joseph J Lim, Edward Johns, Norman Di Palo, Freek Stulp, Antonin Raffin, Samuel Bustamante, João Silvério, Abhishek Padalkar, Jan Peters, Bernhard Schölkopf, Dieter Büchler, Jan Schneider, Simon Guist, Jiajun Wu, Stephen Tian, Haochen Shi, Yunzhu Li, Yixuan Wang, Mingtong Zhang, Heni Ben Amor, Yifan Zhou, Keyvan Majd, Lionel Ott, Giulio Schiavi, Roberto Martín-Martín, Rutav Shah, Yonatan Bisk, Jeffrey T Bingham, Tianhe Yu, Vidhi Jain, Ted Xiao, Karol Hausman, Christine Chan, Alexander Herzog, Zhuo Xu, Sean Kirmani, Vincent Vanhoucke, Ryan Julian, Lisa Lee, Tianli Ding, Yevgen Chebotar, Jie Tan, Jacky Liang, Igor Mordatch, Kanishka Rao, Yao Lu, Keerthana Gopalakrishnan, Stefan Welker, Nikhil J Joshi, Coline Manon Devin, Alex Irpan, Sherry Moore, Ayzaan Wahid, Jialin Wu, Xi Chen, Paul Wohlhart, Alex Bewley, Wenxuan Zhou, Isabel Leal, Dmitry Kalashnikov, Pannag R Sanketi, Chuyuan Fu, Ying Xu, Sichun Xu, brian ichter, Jasmine Hsu, Peng Xu, Anthony Brohan, Pierre Sermanet, Nicolas Heess, Michael Ahn, Rafael Rafailov, Acorn Pooley, Kendra Byrne, Todor Davchev, Kenneth Oslund, Stefan Schaal, Ajinkya Jain, Keegan Go, Fei Xia, Jonathan Tompson, Travis Armstrong, and Danny Driess.  Open x-embodiment: Robotic learning datasets and RT-x models.  In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023.
  * Walke et al. [2023] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al.  Bridgedata v2: A dataset for robot learning at scale.  In Conference on Robot Learning, pages 1723–1736. PMLR, 2023.
  * Wang et al. [2023] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  Cogvlm: Visual expert for pretrained language models.  arXiv preprint arXiv:2311.03079, 2023.
  * Wu and Xie [2023] Penghao Wu and Saining Xie.  V*: Guided visual search as a core mechanism in multimodal llms, 2023.
  * Wu et al. [2023] Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E. Gonzalez, and Trevor Darrell.  See, say, and segment: Teaching lmms to overcome false premises, 2023.
  * Xu et al. [2022] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin.  Discriminator-weighted offline imitation learning from suboptimal demonstrations.  In International Conference on Machine Learning, pages 24725–24742. PMLR, 2022.
  * Yan et al. [2024] Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li.  Vigor: Improving visual grounding of large vision language models with fine-grained reward modeling, 2024.
  * Yang et al. [2024] Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel.  Learning interactive real-world simulators.  In The Twelfth International Conference on Learning Representations, 2024.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  * Yu et al. [2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016.
  * Zhang et al. [2023] Wenjia Zhang, Haoran Xu, Haoyi Niu, Peng Cheng, Ming Li, Heming Zhang, Guyue Zhou, and Xianyuan Zhan.  Discriminator-guided model-based offline imitation learning.  In Conference on Robot Learning, pages 1266–1276. PMLR, 2023.
  * Zhao et al. [2023] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.  Learning fine-grained bimanual manipulation with low-cost hardware.  arXiv preprint arXiv:2304.13705, 2023.
  * Zheng et al. [2024] Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu.  Safe offline reinforcement learning with feasibility-guided diffusion model.  arXiv preprint arXiv:2401.10700, 2024.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In CVPR, 2017.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.

##  Appendix A Limitation and Future Work

Here, we discuss our limitations, potential solutions and interesting future
works.

  1. 1.

Computational Overhead. Note that IVM introduces additional parameters and
computational overhead to directly enhance visual grounding ability of LMMs,
which in turn indirectly improve the VQA performances. However, more VQA
performance gains can be obtained if the same amount of additional parameters
are end-to-end trained directly on VQA data (LLaVA-13B v.s IVM-Enhanced
LLaVa-7B in Table 2).

Solution and future work: Nevertheless, this is quite reasonable because IVM
primarily focuses on improving the visual grounding ability, but accurate VQA
also requires other abilities which can be learned through end-to-end
training. End-to-end training, however, requires tremendous VQA data to
implicitly and slowly improve the visual grounding ability, which is quite
data-intensive. Both Table 4.1 and Figure 2 can show that even trained on
billions of data, GPT4-V still performs subpar on tasks that require strong
visual grounding ability. IVM, instead, can significantly boost the visual
grounding ability of GPT4-V using just 7B parameters and less computations.
One promising and interesting future direction is to include some auxiliary
tasks to directly absorb the strong visual grounding ability in the IVM-Mix-1M
dataset through end-to-end training like [59].

  2. 2.

Data Quality. Due to task complexity, the machine-annotated data in IVM-Mix-1M
inevitably includes wrong labels that mistakenly exclude instruction-sensitive
image regions or suboptimal labels that not fully mask out all instruction-
irrelevant areas. These inaccuracies may lead to suboptimal IVM model. We
propose a DWSL framework to tackle this. However, the DWSL framework relies on
a learned discriminator and a human-designed $f(x)$ function, which may not
exclude all inaccuracies.

Solution and future work: We have clearly demonstrated in Figure 9 that with a
simple $f(x)$ and a lightweight discriminator, DWSL consistently outperforms
the naive Supervised Learning (SL), doing pretty well on prioritizing good
samples and meanwhile identifying inaccurate labels. To further enhance this,
one can use other advanced techniques such as Reinforcement Learning from
Human Feedback (RLHF) [46, 5, 26] to provide more fine-grained judgement on
annotation qualities, or resort a theoretical-soundness $f(x)$ [60] to achieve
better results. In addition, one can also use our pretrained IVM model to
directly generate high-quality heatmaps to enhance the machine annotations.

  3. 3.

Mask Deployment Methods. In this paper, we directly use the simple post-
processing method to apply the IVM generated heatmaps on images, which then
are fed into LMMs to perform downstream tasks. However, these post-processing
methods introduce some heuristics, which may be suboptimal for downstream
LMMs. In addition, LMMs may not see many masked images during pretraining,
thus some distributional shift may occur.

Solution and future work: Although these limitations exist, IVM still obtain
consistent improvements using diverse mask deployment methods, as shown in
Table 10, which showcases the great versatility of IVM to inject visual
grounding abilities. To further improve this, one strategy is employ some
task-specialized visual search method [58], but will bring many computational
load during inference and limit the versatility on embodied agents. Another
promising direction is directly using the IVM generated heatmaps as an
additional input channel to finetune the LMMs like [52], which can fully
eliminate the heuristics of post-process methods, may bring larger performance
gains. Due to resources limits, we leave this for a future work.

  4. 4.

Fine-grained Heatmaps. Note that the IVM generated heatmaps cannot provide
exact semantic object segmentation with clear contours like reasoning
segmentation [31] offers.

Discussions: We want to clarify that this is an advantage of the IVM model
rather than a limitation. This is because of the ambiguous nature of the
visual masking task. For this task, the ground truth heatmaps are mostly less
semantic-meaningful for annotations as discussed in 3.1. So, we ensemble the
annotation proposals from different visual grounding methods for data
annotation, which will make the trained IVM model robust to include
instruction-relevant image areas, rather than being aggressive to exclude some
instruction-sensitive pixels like reasoning segmentation [31] does illustrated
in Figure 2.

Overall, although some limitation exist, we have thoroughly discussed
potential solutions to these limitations. Moreover, in this paper, we have
demonstrated the superior effectiveness and versatility of IVM to directly
inject strong visual grounding ability to downstream LMMs or embodied agents,
representing a pioneer effort to extend traditional visual grounding methods
towards a more complex and generic setting that covers diverse multimodal
instruction following tasks.

##  Appendix B Broader Impact

This paper aims to advance the field of artificial intelligence, where no
significant negative social impact is observed in this paper. The IVM-Mix-1M
may contain some potential privacy issues and biases. However, in this paper,
nearly all data are collected from open-sourced data, which have been well
peer-reviewed, thus resolved this ethical concern.

##  Appendix C Training and Evaluation Details

###  C.1 Architecture Details

In this section, we primarily focus on the architectural design of the
lightweight generator and discriminator, as both the Language Model Multitask
(LMM) and the vision backbone are derived from the powerful foundation models
(LLaVA & SAM). Both the generator and discriminator utilize the same
transformer-based decoder block, as depicted in Figure 11. We employ two such
blocks for both the generator and discriminator. Specifically, the generator
produces dense predictions by upscaling the output features of the decoder
block through a straightforward upsampling operation. In contrast, the
discriminator first employs a two-layer convolutional downsampling network to
encode segmentation labels. This network, in conjunction with the decoder
block and a simple MLP (multi-layer perceptron) head, outputs the weights.

Figure 11: Generator/Discriminator Architecture Details

###  C.2 Training Details

We adopt 8 NVIDIA 80G A100 GPUs and take 4 days to train our IVM model. The
training scripts are based on deepspeed [4] engine and the training
hyperparameters can be found in Table 4.

Table 4: Hyper-parameters for pretraining. config | value
---|---
training iteration | 200K
optimizer | AdamW [43]
learning rate | $1\times 10^{-5}$
batch size | 32
weight decay | 0
optimizer momentum |  $\beta_{1},\beta_{2}$=0.9, 0.95
data augmentation | RandomCropResize

###  C.3 Multimodal Benchmarks Evaluations

We evaluate our IVM on diverse multimodal benchmarks, including general VQA
(VQAv2 [18], GQA [27], MME [16]), first-person perspective QA (EgoThink [10]),
scientific QA (SQA [44]), hallucination adversarial QA (POPE [34]) and V*
[58], a recently proposed challenging benchmark with high-resolution and
complex visual input.

Our evaluation employs a two-stage inference pipeline: the image is firstly
simplified by IVM-generated heatmap and mask deployment methods; Subsequently,
the simplified image is fed into downstream LMMs(GPT4-V [1], LLaVA [38])
without finetuning. We adhere to the official procedures of each benchmark to
evaluate the output of LMMs and report the results.

###  C.4 Real Robot Evaluations

Task descriptions. The real robot experiments evaluate several pick and place
manipulation tasks that require strong visual grounding abilities.
Specifically, we evaluate on 4 tasks as shown in Table 5, following the task
definitions in DecisionNCE [35]. For each task, we collect around 100
demonstrations using the demonstration collection system in BridgedataV2 [56].
We take both a side camera view and a wrist camera view as the vision inputs,
as shown in Figure 12. For each demonstration, the environmental steps are
around 50 steps. During data collection, the object and robot locations are
randomly initialized, and the scene also has lots of randomly located
distractors with varied shape and color.

Figure 12: Visual input view for LCBC policy. Table 5: Real Robot Tasks Environment ID | Language Instruction
---|---
Red cup on silver pan | Pick up the red cup and place it on the silver pan
Red cup on red plate | Pick up the red cup and place it on the red plate
Duck on green plate | Pick up the duck and place it on the green plate
Duck in pot | Pick up the duck and place it in the pot

Training details. Here, we train Language-Conditioned Behavior Cloning (LCBC)
policies using DDPM [24] loss since diffusion policies are good at fitting
complex data distributions [67, 56, 2], especially human demonstrations [11].
For model architecture, the side and wrist images are augmented and then
passed through a shared ResNet50 [22] image encoder and get an image embedding
for each camera view, following [56]. As the downstream data is quite limited,
we load the ImageNet [13]-pretrained ResNet50 image encoder and further train
it on the small robot data. Meanwhile, the language instruction is passed
through a frozen T5 text-encoder [48], which is fused into the image encoder
via Film conditioning layers [47]. Then, this language-conditioned image
embedding is passed through a MLPs with residual connections similar to IDQL
[21], which then outputs the predicted noise in DDPM [24]. To obtain smoothed
policy rollouts, we adopt Action Chunking and Temporal Ensemble from ACT [66]
with a chunking size 4 rather than 100 in [66] because the episode horizons in
this paper are only around 50. The LCBC policies are trained either on the
original side camera view (without IVM assistance) or on the IVM-masked side
camera view (with IVM assistance) for 200K steps with a batch size of 64. The
training can be completed on 2 NVIDIA RTX4090 GPU in 17h. All hyperparameters
are summarized in Table 6.

Table 6: Real robot LCBC training details Backbones
---
Visual encoder | Resnet50 [22] (ImageNet [13] pretrained)
text encoder | T5 [48] (frozen)
DDPM hyperparameters
noise schedule | VP [51]
denoising time steps | 25
Other hyperparameters
Chunking size | 4
Optimizer | AdamW [43]
Learning rate | 1e-4
Lr schedule | cosine annealing
Warm up steps | 2000
Batch size | 64
Gradient Steps | 200K
Augmentation | Yes [56]

Evaluation details. We first evaluate the trained LCBC policies without strong
distractions, where no or only small distractors appear in the image. Then, we
add lots of distracting objects with varied shapes and colors, and even
introduce strong human disturbance to attack the LCBC policies. For each score
reported in Figure 8, we evaluate 10 episodes and report the success rates.

##  Appendix D Mixture of Expert Annotation Pipeline

###  D.1 Labeled Visual Grounding data

For labeled visual grounding data, We provide the following prompt to drive
GPT-4 [1] to generate more complex instructions based on given language
annotations.

[Image Description] %s [System] You are an AI visual assistant, and you are
seeing a single image. What you see are part of the image and are provided
with a simple phrase. Please generate any instructions that can be executed
based on the content of the picture described, including simple queries about
the content of the picture, such as the object types, counting the objects,
object actions, relative positions between objects, etc. Also consider more
complex questions that require reasoning. For example, you can ask what time
it is now for a clock and what can I use to clean the room for a broom. Ensure
that the questions you ask can be clearly answered only based on what you see.
Please generate as many five questions as possible and return them in a single
line separated by ’;’ and avoid any other output.

###  D.2 Unlabeled Visual Instruction Following Data

For unlabeled visual instruction following data, we first try to simplify
complex instructions. Specifically, we employ GPT-4 to infer the necessary
object for executing the given instructions based on these instructions and a
simple image caption. If the dataset lacks captions, they can be generated
using an existing caption model like BLIP-Caption [33]. Below, we outline the
prompts specifically designed for GPT-4.

[Image Caption] %s [Instruction] %s [System] You are an helpful AI assistant.
I need to reply to the previous instruction based on an image, and I have a
simple caption for the image. Please note that there may be objects in the
image that I did not detect. Since you cannot view the image, please list any
potential objects that might influence my responses, separated by semicolons,
in a single line without any additional output. If you believe that the number
of objects could be too extensive and might hinder my judgment, print ’None’.

With the simplified instruction, we can adopt existing visual grounding models
to generate the candidate label. Specifically, we utilize four models:
AlphaCLIP [52], LISA [31], OwVIT [20] and Grounding-SAM [49] and the inference
pipelines are provided in the official implementation of these models.

##  Appendix E More result

###  E.1 Referring Expression Comprehension

As IVM is an extension of traditional visual grounding task, we also evaluate
our IVM on RefCoCo, RefCoCo+ and RefCoCog [64]. We reported the accuracy
(IOU-50%) on the validation split in Table 7. As a generalist model capable of
handling complex instructions, our IVM achieves performance comparable to that
of state-of-the-art (SOTA) specialist models.

Table 7: result in REC Methods | RefCoCo | RefCoCo+ | RefCoCog
---|---|---|---
Specialist models
G-DINO-L [39] | 90.56 | 82.75 | 86.13
Generalist models
LLaVA-7B [41] | 76.29 | 66.76 | 70.4
IVM(Ours) | 90.1 | 83.3 | 82.9

###  E.2 Visualization Result

In this section, we provide more visualization result in VQA-type data as
shown in Figure 13.

Figure 13: Visualization results of IVM generated masks.

Failure Case. Although we observe numerous successful instances, our IVM still
faces significant challenges, as illustrated in Figure 14. We summarize these
challenges into three categories: missing target, misguided target, and
insufficient reasoning.

(a) Missing Target: Challenges arise when target objects are relatively small
and scattered around many separate image corners. In this case, accurately
detecting all of the targeted objects is quite difficult. Even specialized
open vocabulary detection models struggle with this task. For example, the cup
on the right in the image is masked by the IVM mistakenly. However, we still
observe that the IVM-generated heatmap for the right cup is partially
activated, meaning that IVM have partially focus this regions. We believe by
providing more training data, IVM can handle this better.

(b) Misguided Target: Accurately Localizing tiny target objects is a
recognized challenge [58], especially when similar but more obvious objects
are present. For instance, IVM incorrectly focuses on the more centrally
located shoes of another man, instead of the shoes of the man wearing the red
hat at the edge of the picture. However, this instruction is pretty
challenging that at first glance, even a human might struggle to spot the man
with the red hat in the left corner. We will leave challenge scenarios like
this for future research.

(c) Insufficient reasoning: The objective of the IVM task is to assist LMMs in
extracting visual features more effectively to better follow instructions.
Thus, the demands on the model’s reasoning capabilities extend far beyond mere
object localization. Although IVM demonstrates strong performance, it
sometimes overlooks additional image content necessary for accurately
following instructions after correctly locating the target object. For
instance, while IVM successfully identified the braking motorcycle, it failed
to recognize that answering the question requires knowledge of the positions
of both motorcycles simultaneously. We attribute this issue to biases in the
training data. By incorporating more complex instructions and diversified
labels, we anticipate that our model will achieve improved performance

Figure 14: Some failure cases.

###  E.3 Robotics Result

Here, we provide more evaluation rollouts of the IVM-assisted LCBC agents
under strong distractions. Figure 15 clearly demonstrates that even under
strong distractions like the background are full of distracting objects with
similar colors or shapes to the targeted objects, and strong human
disturbances that adversarially attack the robots, the IVM-assisted LCBC
agents can still complete the tasks pretty well, enjoying high-level of
generalization and robustness thanks to the superior visual grounding ability
injected by IVM. More videos can be found in the supplementary materials.

Figure 15: Real robot LCBC results with IVM assistance.


# Residual Chain Prediction for Autonomous Driving Path Planning

Liguo Zhou∗, Yirui Zhou∗, Huaming Liu, Alois Knoll Liguo Zhou, Yirui Zhou, and
Alois Knoll are with the Chair of Robotics, Artificial Intelligence and Real-
time Systems, Technical University of MunichLiguo Zhou is with the School of
Computer Science, Nanjing University of Information Science &
TechnologyHuaming Liu is with the School of Computer and Information
Engineering, Fuyang Normal University* These authors contributed equally to
this work.

###### Abstract

In the rapidly evolving field of autonomous driving systems, the refinement of
path planning algorithms is paramount for navigating vehicles through dynamic
environments, particularly in complex urban scenarios. Traditional path
planning algorithms, which are heavily reliant on static rules and manually
defined parameters, often fall short in such contexts, highlighting the need
for more adaptive, learning-based approaches. Among these, behavior cloning
emerges as a noteworthy strategy for its simplicity and efficiency, especially
within the realm of end-to-end path planning. However, behavior cloning faces
challenges, such as covariate shift when employing traditional Manhattan
distance as the metric. Addressing this, our study introduces the novel
concept of Residual Chain Loss. Residual Chain Loss dynamically adjusts the
loss calculation process to enhance the temporal dependency and accuracy of
predicted path points, significantly improving the model’s performance without
additional computational overhead. Through testing on the nuScenes dataset, we
underscore the method’s substantial advancements in addressing covariate
shift, facilitating dynamic loss adjustments, and ensuring seamless
integration with end-to-end path planning frameworks. Our findings highlight
the potential of Residual Chain Loss to revolutionize planning component of
autonomous driving systems, marking a significant step forward in the quest
for level 5 autonomous driving system.

###### Index Terms:

Path planning, imitation learning, behavior cloning, covariate shift, distance
metrics

##  I Introduction

Autonomous driving technologies have significantly advanced the quest for
fully autonomous vehicles, revolutionizing transportation systems and
mobility. Traditional autonomous driving systems utilize a modular deployment
strategy, categorized into three core functionalities: perception, prediction,
and planning [1]. Which are called Modular Pipelines. At the heart of modular
pipelines is the pursuit of developing reliable and efficient path planning
algorithms, which are crucial for ensuring that vehicles navigate through
complex environments safely.

Historically, path planning methodologies have relied on intricate rule-based
designs such as Dijkstra and A* Search Algorithm [2] or sampling-based method
such as Rapidly exploring Random Tree (RRT) and Probabilistic Road Map (PRM).
Despite their effectiveness in static environments, often fall short in
dynamic and unpredictable urban situations, due to the fact that they rely on
manually defined costs and reward functions that could lead to optimal driving
behaviors. Accordingly, there is a discernible shift in the field towards
leveraging neural networks as a foundational basis for the evolution of path
planning strategies, with a particular emphasis on the accelerated advancement
of end-to-end path planning methodologies. These learning-based approaches
promise improvements in adaptability and performance with the acquisition of
larger dataset [3, 4, 5]. End-to-end path learning simplifies the design and
development process of the whole system. It reduces the need for manually
designed features extraction and preprocessing steps, as the model can learn
to extract useful features directly from the data. Learning-based path
planning methods have the potential to significantly enhance AD system’s
ability to understand and interact with its environment.

In the realm of end-to-end path planning, a sequence of future trajectory
points is deduced from either raw sensor data or preprocessed data acquired
from the vehicle sensors. Conventionally, predicted path points facilitate a
direct comparison with expert navigated path points, utilizing distance
metrics such as the Manhattan distance (L1 norm) or Euclidean distance (L2
norm), commonly employed within the scope of imitation learning. This
methodology, however, presents notable challenges, including difficulties in
model convergence and loss in generalization capability. To mitigate these
issues, a transition from global to relative coordinates—measured in relation
to the initial point—is proposed. Furthermore, additional methodologies such
as the Frechet Distance, Chamfer Distance, and Hausdorff Distance, provide
alternative means of calculating the discrepancies between predicted and
expert-navigated path points. While these methods offer unique perspectives in
measuring discrepancies, they fall short in addressing the crucial component
of temporal dependencies inherent in dynamic driving environments. Dynamic
Time Warping (DTW) presents a methodology for calculating the discrepancy
between two temporally misaligned sequences, embodying an approach that
contemplates the entire sequence of data points [6]. However, DTW lacks in its
provision for adjustments to future points predicated on their pre-predicted
path points. These methods also face the problem of covariate shift [7].

To rectify this shortfall, we propose the innovative concept of Residual Chain
Loss. This approach is designed to enhance both the accuracy and robustness of
autonomous driving path planning by introducing a mechanism for dynamic loss
adjustment without increasing calculation cost. This adaptation ensures an
incorporation with a temporal dimension that reflects the sequential nature of
driving decisions, closely mirroring the complex dynamics of real-world
driving scenarios. To sum up, the key contribution of this work include:

  * •

Enhanced Handling of Covariate Shift: By adopting relative coordinates as
outputs, our method facilitates a more effective alignment between expert and
predicted points within the framework of imitation learning. This strategic
choice also offers a robust solution to the problem of covariate shift.

  * •

Dynamic Loss Adjustment: Introduction of a dynamic loss adjustment mechanism
enables the model to adapt its learning process based on the sequence of
previously predicted points, enhancing model accuracy over time without
increasing much computation time.

  * •

Compatibility with End-to-End Path Planning Models: Our approach is designed
to be compatible with a variety of learning-based models, facilitating its
adoption and integration into a broad spectrum of path planning model.

##  II Related Work

Trajectory-Based Methods. Trajectory-based Methods could be classified as
Traditional algorithm of map-based path planning algorithms and sampling-based
path planning algorithm. Most commonly used traditional algorithms could be
defined as A-Class [8]. Introduced by E.W. Dijkstra in 1959 [9], Dijkstra
Algorithm employs a greedy approach to iteratively explore as many subnodes as
possible, subsequently utilizing the technique of relaxation to refine and
optimize the choice of paths. A* Algorithm is the most commonly used method,
improved on Dijsktra Algorithm, A* Algorithm introduces the heuristic
function, which makes the algorithm more efficient than Dijkstra Algorithm.
Most commonly applied sampling-based methods are Rapidly exploring Random Tree
(RRT) [10] and Probabilistic Road Map (PRM) [11]. These algorithms rely on
human engineering rather than on data to achieve functionalities. As a result,
while they exhibit robust performance in static environments, their ability to
navigate complex dynamic environments remains limited.

End-to-end Path Planning. End-to-end path planning represents a paradigm shift
from traditional modular pipelines, which distinctly segregate perception,
prediction, and planning modules. This holistic approach not only simplifies
the integration of these key components but also facilitates comprehensive
optimization, potentially reducing computational costs [12]. End-to-end path
planning methods could be classified as two streams: Imitation Learning and
Reinforcement Learning. Within the domain of Imitation Learning, two
subcategories emerge, namely Behavior Cloning (BC) [13] and Reverse
Reinforcement Learning (RIL) [14]. Contrary to Behavior Cloning (BC), which
directly mimics expert demonstrations to predict actions, Inverse
Reinforcement Learning (IRL) deduces the underlying reward function from
expert demonstrations. This approach divides into two key tasks: deriving the
cost function through end-to-end learning and optimizing trajectory sampling
efficiently [12]. IRL offers a deeper insight into expert decisions by
understanding their motivations. Reinforcement Learning is the learning of
trial. It requires more data in comparison with supervised learning, thus
almost all ADS based on RL are investigated in simulation [12]. Up to now, RL
has successfully work on a car driving on an empty lane on the street, but IL
has done it 3 decade before. In the field of end-to-end path planning, RL
still has a long way chasing after IL. Recently, there are some works
combining supervised learning with RL. Implicit affordances [15] and GRI [16]
pretrain CNN encoder using auxilary tasks such as object detection, semantic
segmentation and classification. The image encoder is frozen, policy is
trained on implicit affordance of outputs of the frozen image encoder [17].

Imitation Learning (IL) and Behavior Cloning (BC). Initially, Behavior Cloning
was applied to end-to-end path planning in [18], setting a precedent for
subsequent advancements. Progressions in this field have seen the
incorporation of multi-sensor fusion [19], auxiliary tasks [20], and improved
expert-design [21], each contributing to the enhanced performance of BC in the
realm of autonomous navigation. However, despite its considerable successes
across a range of applications, BC encounters intrinsic limitations,
particularly when faced with the challenge of capturing long-tail sequences
that are prevalent in real-world scenarios [22]. The algorithm has already
applied on roads [23], including urban driving and highways. In our work, we
employ an imitation learning approach, sampling specific points along with
their past and future points and their position information. By treating the
future points as the expert, we fit the predicted path points based on past
points to these future positions. This method allows us to leverage the
accuracy of future data as a guiding benchmark for training our model to
predict the trajectory more accurately, bridging the gap between past
observations and future states.

Distance Metrics. In the realms of imitation learning and behavior cloning,
our primary objective centers on refining the model’s outputs to more closely
align with the expert’s behavior. This endeavor becomes particularly salient
within the purview of path planning, where the choice of a distance function
between ground truth and predicted path points emerges as a pivotal
consideration. Traditional methodologies employ Manhattan and Euclidean
distances, yet alternative metrics such as Frechet, Chamfer, and Hausdorff
distances present nuanced variations for evaluation. Furthermore, Dynamic Time
Warping (DTW) [6] offers an advanced framework for assessing the distance
between two temporally misaligned sequences. These conventional approaches
applied in BC, predicated upon deriving full future path point predictions
from a given point and its past points, are prone to incremental deviations
from the expert’s path. BC treats each points independently [12]. Thereby
necessitating the covariate shift [7], several on-policy methods have been
proposed to address this issue [24] [25]. Other than that, Data as
Demonstrator (DAD) [26] proposes an innovative solution by pairing predicted
point with the next expert point in the sequence. We depart from the
conventional paradigm of predicting the coordinates of future points. Instead,
our outputs are formulated as relative coordinates between adjacent path
points in the sequence. This adjustment is accomplished by modifying the
ground truth to represent the Manhattan distance between the expert
coordinates and the sum of the initial point coordinates and the cumulative
relative coordinates between adjacent pre-predicted points in the sequence,
which are the pre-predicted outputs. Such a loss function meticulously
accounts for the dynamic adjustments predicated on previous predictions while
simultaneously assessing the relative displacement between predicted and
expert points.

##  III Method

###  III-A End-to-End Driving Model

The driving process of a human driver can be described as Equation 1, where
$X$ is what the driver sees and hears, $Y$ is the driver’s operations to the
vehicle, and $f$ is the driver’s nervous system that has been trained in
driving school and past driving experience. If we are passengers, we can see
$X$ and $Y$, but we can’t see the processing in $f$ because that’s in the
driver’s nervous system.

$Y=f(X).$ (1)

Similar to human drivers, the end-to-end machine learning model of autonomous
driving has two ends for inputting $X$ and outputting $Y$, and all the other
processes are done unexplainably inside the model $f$, such as the feed-
forward of the deep neural network. In the extreme case, the input $X$ of the
end-to-end model is the raw data of the sensors, and the output $Y$ is the
operating commands to the vehicle. In most cases, the input $X$ is
preprocessed sensor data, and the output $Y$ is a planned path in the future.
The planned path, which is denoted by a series of positions, is then converted
into vehicle control commands through the control algorithms.

Figure 1: Learning a Path by Imitation.

###  III-B Path Planning based on Imitation Learning

Figure 1 shows the process of learning a path by imitation:

  * 1

Sampling historical driving paths at equal time intervals. Each sampled path
point includes the vehicle’s position and rotation, which denotes the vehicle
pose. Each point also has corresponding environment data, such as images and
point clouds, and vehicle status, such as throttle, wheel angle, and braking
status.

  * 2

Select a sampled point as the current point and split the other sampled points
into past and future. The point in the ”$\square$” denotes the current pose.
The points before it belong to the past, and the points behind it belong to
the future. The current point and $n$ consecutive points before it, as well as
their corresponding environment data and vehicle statuses, are the input of
the path planning model. The $m$ consecutive points after the current point
are the ground truth of the predicted path points of the model.

  * 3

Fitting the predicted path points to the ground truth. At the beginning of
training the path planning model, the predicted path points are random values.
In the training iterations, the predicted path points become closer and closer
to the ground truth by optimizing the model parameters.

###  III-C Loss Function

In deep learning, optimizing a model’s parameters is guided by stochastic
gradient descent (SGD) with a loss function. The goal of the SGD is to make
the result of the loss function as small as possible.

Generally, the Manhattan distance (L1 norm) and Euclidean distance (L2 norm)
are two commonly used loss functions to measure the difference between two
series of path points. Given the ground truth positions $(x_{t},y_{t})$ and
the predicted positions $(x_{t}^{\prime},y_{t}^{\prime})$ where $x$ and $y$
are the horizontal and vertical coordinates and $t=(1,2,3,...,m)$, the L1 and
L2 loss function are defined as Equations 2 and 3,

$\begin{split}l=\frac{1}{2m}\sum_{i=1}^{m}(|{x_{t}-x_{t}^{\prime}}|+|{y_{t}-y_{t}^{\prime}}|),\end{split}$
(2)

$\begin{split}l=\frac{1}{2m}\sum_{i=1}^{m}[({x_{t}-x_{t}^{\prime}})^{2}+({y_{t}-y_{t}^{\prime}})^{2}],\end{split}$
(3)

where $l$ is the loss value. Since vehicles generally drive on the road, the
vertical coordinate $z$ is ignored here to simplify the description.

###  III-D Coordinate Transformation

In practice, the global coordinates $x$ and $y$ have no fixed range. If we
directly use the original coordinate values to train a deep neural network, it
will be difficult to converge, and the model will lose generalization.
Therefore, we need to convert the global coordinates into relative coordinates
by Equation 4 where $(x_{0},y_{0})$ is the current position.

$\begin{split}x_{t}=x_{t}-x_{0},\quad y_{t}=y_{t}-y_{0}.\end{split}$ (4)

TABLE I: Coordinate Conversion Comparison | Original Coordinate
---|---
|  Current | Future
|  $x_{0}$ |  $x_{1}$ |  $x_{2}$ |  $x_{3}$ |  … |  $x_{m}$
|  $y_{0}$ |  $y_{1}$ |  $y_{2}$ |  $y_{3}$ |  … |  $y_{m}$
Conversion Method |  |  Model’s Target (Ground Truth) |
|  |  $\Delta x_{1}$ |  $\Delta x_{2}$ |  $\Delta x_{3}$ |  … |  $\Delta x_{m}$
|  |  $\Delta y_{1}$ |  $\Delta y_{2}$ |  $\Delta y_{3}$ |  … |  $\Delta y_{m}$
Equation 4 |  |  $x_{1}-x_{0}$ |  $x_{2}-x_{0}$ |  $x_{3}-x_{0}$ |  … |  $x_{m}-x_{0}$
|  |  $y_{1}-y_{0}$ |  $y_{2}-y_{0}$ |  $y_{3}-y_{0}$ |  … |  $y_{m}-y_{0}$
Equation 5 |  |  $x_{1}-x_{0}$ |  $x_{2}-x_{1}$ |  $x_{3}-x_{2}$ |  … |  $x_{m}-x_{m-1}$
|  |  $y_{1}-y_{0}$ |  $y_{2}-y_{1}$ |  $y_{3}-y_{2}$ |  … |  $y_{m}-y_{m-1}$
Equation 8 |  |  $x_{1}-x_{0}$ |  $x_{2}-(x_{0}+\Delta x_{1}^{\prime})$ |  $x_{3}-(x_{0}+\Delta x_{1}^{\prime}+\Delta x_{2}^{\prime})$ |  … |  $x_{m}-(x_{0}+\sum_{i=1}^{m-1}\Delta x_{i}^{\prime})$
|  |  $y_{1}-y_{0}$ |  $y_{2}-(y_{0}+\Delta y_{1}^{\prime})$ |  $y_{3}-(y_{0}+\Delta y_{1}^{\prime}+\Delta y_{2}^{\prime})$ |  … |  $y_{m}-(y_{0}+\sum_{i=1}^{m-1}\Delta y_{i}^{\prime}))$
|  | Model’s Output |
|  |  $\Delta x_{1}^{\prime}$ |  $\Delta x_{2}^{\prime}$ |  $\Delta x_{3}^{\prime}$ |  … |  $\Delta x_{m}^{\prime}$
|  |  $\Delta y_{1}^{\prime}$ |  $\Delta y_{2}^{\prime}$ |  $\Delta y_{3}^{\prime}$ |  … |  $\Delta y_{m}^{\prime}$

In this work, we further convert the relative coordinates that can make the
performance of the model better. First, we try to convert each coordinate into
an increment relative to its previous coordinate, as shown in Equation 5.

$\Delta x_{t}=x_{t}-x_{t-1},\quad\Delta y_{t}=y_{t}-y_{t-1}.$ (5)

In training, the model should output $\\{(\Delta x_{t}^{\prime},\Delta
y_{t}^{\prime})|t=(1,2,3,...,m)\\}$ and use Equation 2 or Equation 3 to
compute the loss between it and $\\{(\Delta x_{t},\Delta
y_{t})|t=(1,2,3,...,m)\\}$. However, in testing, we can not just simply
generate the predicted path points
$\\{(x_{t}^{\prime},y_{t}^{\prime})|t=(1,2,3,...,m)\\}$ by Equation 6

$x_{t}^{\prime}=x_{t-1}+\Delta x_{t}^{\prime},\quad
y_{t}^{\prime}=y_{t-1}+\Delta y_{t}^{\prime},$ (6)

because $x_{t-1}$ and $y_{t-1}$ are unknown in testing. The correct
calculation method is

$\begin{split}x_{t}^{\prime}=x_{0}+\sum_{i=1}^{t}\Delta x_{i}^{\prime},\quad
y_{t}^{\prime}=y_{0}+\sum_{i=1}^{t}\Delta y_{i}^{\prime}.\end{split}$ (7)

Now, we can find that, except for the $(\Delta x_{1}^{\prime},\Delta
y_{1}^{\prime})$, in training, the target of $(\Delta x_{t}^{\prime},\Delta
y_{t}^{\prime})$ is $(x_{t}-x_{t-1},y_{t}-y_{t-1})$, but in testing, the
target is $[x_{t}-(x_{0}+\sum_{i=1}^{t-1}\Delta
x_{i}^{\prime}),y_{t}-(y_{0}+\sum_{i=1}^{t-1}\Delta y_{i}^{\prime})]$.
Therefore, in order to achieve better results in testing, we convert the
Equation 5 into Equation 8,

$\begin{split}\Delta x_{t}=\begin{cases}x_{1}-x_{0}&\text{if }t=1,\\\
x_{t}-(x_{0}+\sum_{i=1}^{t}\Delta
x_{i}^{\prime})&\text{otherwise},\end{cases}\\\ \Delta
y_{t}=\begin{cases}y_{1}-y_{0}&\text{if }t=1,\\\
y_{t}-(y_{0}+\sum_{i=1}^{t}\Delta
y_{i}^{\prime})&\text{otherwise},\end{cases}\end{split}$ (8)

which means in training iterations, the ground truth needs to be dynamically
adjusted according to the output of the model.

In order to visually compare different conversion methods, we list the results
of each conversion method in Table I.

##  IV Experiment

In this section, we train an end-to-end path planning model on nuScenes [27]
dataset. In the training phase, we use $\frac{1}{2m}\sum_{i=1}^{m}[({\Delta
x_{t}-\Delta x_{t}^{\prime}})^{2}+({\Delta y_{t}-\Delta y_{t}^{\prime}})^{2}]$
to guide the network optimization. For comparison, we recover the $(\Delta
x_{t}^{\prime},\Delta y_{t}^{\prime})$ to $(x_{t},y_{t})$ to calculate both
training loss and validation loss. Figure 2 shows the change of loss for two
coordinate transform methods. The left one uses the method of Equation 4, and
the right one uses our method of Equation 8. Compared with the other method,
our method reduces the loss by 15%.

Figure 2: Loss Comparison

##  V Conclusion

This paper presented the Residual Chain Loss method, a novel approach designed
to address the challenges of path planning and behavior cloning for autonomous
driving system. Our model outputs are formulated as relative coordinates
between adjacent path points in the sequence. This adjustment is accomplished
by modifying the ground truth to represent the Manhattan distance between the
expert coordinates and the sum of the initial point coordinates and the
cumulative pre-predicted outputs, in order to achieve dynamic loss adjustment.
Thus our method effectively mitigates the issues of covariate shift and
enhances model performance without increasing computational cost. Our
experiments on the nuScene dataset validate the approach’s compatibility with
end-to-end path planning models, indicating substantial improvements over
traditional relative coordinates methods. The results affirm the potential of
Residual Chain Loss to advance the development of more efficient and reliable
autonomous driving systems. Future work will explore its

##  VI Acknowledgement

## References

  * [1] S. Teng, X. Hu, P. Deng, B. Li, Y. Li, Y. Ai, D. Yang, L. Li, Z. Xuanyuan, F. Zhu, and L. Chen, “Motion planning for autonomous driving: The state of the art and future perspectives,” IEEE Transactions on Intelligent Vehicles, vol. 8, p. 3692–3711, June 2023.
  * [2] J. Ziegler, M. Werling, and J. Schroder, “Navigating car-like robots in unstructured environments using an obstacle sensitive cost function,” in 2008 IEEE Intelligent Vehicles Symposium, pp. 787–791, 2008.
  * [3] S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,” 2022.
  * [4] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y. Qiao, and H. Li, “Planning-oriented autonomous driving,” 2023.
  * [5] M. Vitelli, Y. Chang, Y. Ye, M. Wołczyk, B. Osiński, M. Niendorf, H. Grimmett, Q. Huang, A. Jain, and P. Ondruska, “Safetynet: Safe planning for real-world self-driving vehicles using machine-learned policies,” 2021.
  * [6] R. Tavenard, “An introduction to dynamic time warping.” https://rtavenar.github.io/blog/dtw.html, 2021.
  * [7] S. Ross, G. J. Gordon, and J. A. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” 2011.
  * [8] Z. Tang and H. Ma, “An overview of path planning algorithms,” IOP Conference Series: Earth and Environmental Science, vol. 804, p. 022024, 07 2021.
  * [9] E. Dijkstra, “A note on two problems in connexion with graphs. numeriske mathematik, 1, 269–271,” 1959.
  * [10] J. Kuffner and S. LaValle, “Rrt-connect: An efficient approach to single-query path planning,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), vol. 2, pp. 995–1001 vol.2, 2000.
  * [11] L. Kavraki, P. Svestka, J.-C. Latombe, and M. Overmars, “Probabilistic roadmaps for path planning in high-dimensional configuration spaces,” IEEE Transactions on Robotics and Automation, vol. 12, no. 4, pp. 566–580, 1996.
  * [12] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li, “End-to-end autonomous driving: Challenges and frontiers,” 2023.
  * [13] M. Bain and C. Sammut, “A framework for behavioural cloning,” in Machine Intelligence 15, Intelligent Agents [St. Catherine’s College, Oxford, July 1995], (GBR), p. 103–129, Oxford University, 1999.
  * [14] A. J. Snoswell, S. P. N. Singh, and N. Ye, “Revisiting maximum entropy inverse reinforcement learning: New perspectives and algorithms,” in 2020 IEEE Symposium Series on Computational Intelligence (SSCI), IEEE, Dec. 2020.
  * [15] M. Toromanoff, E. Wirbel, and F. Moutarde, “End-to-end model-free reinforcement learning for urban driving using implicit affordances,” 2020.
  * [16] R. Chekroun, M. Toromanoff, S. Hornauer, and F. Moutarde, “Gri: General reinforced imitation and its application to vision-based autonomous driving,” 2022.
  * [17] M. Toromanoff, E. Wirbel, and F. Moutarde, “Is deep reinforcement learning really superhuman on atari? leveling the playing field,” 2019.
  * [18] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars,” 2016.
  * [19] D. Chen and P. Krähenbühl, “Learning from all vehicles,” 2022.
  * [20] F. Codevilla, E. Santana, A. M. López, and A. Gaidon, “Exploring the limitations of behavior cloning for autonomous driving,” 2019.
  * [21] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. V. Gool, “End-to-end urban driving by imitating a reinforcement learning coach,” 2021.
  * [22] O. Scheel, L. Bergamini, M. Wołczyk, B. Osiński, and P. Ondruska, “Urban driver: Learning to drive from real-world demonstrations using policy gradients,” 2021.
  * [23] J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, and A. Kendall, “Urban driving with conditional imitation learning,” 2019.
  * [24] K. Judah, A. P. Fern, T. G. Dietterich, and P. Tadepalli, “Active imitation learning: Formal and practical reductions to i.i.d. learning,” Journal of Machine Learning Research, vol. 15, no. 120, pp. 4105–4143, 2014.
  * [25] S. Ross and J. A. Bagnell, “Reinforcement and imitation learning via interactive no-regret learning,” 2014.
  * [26] Arun, M. Hebert, and J. A. Bagnell, “Data as demonstrator with applications to system identification,” 2014.
  * [27] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” arXiv preprint arXiv:1903.11027, 2019.