L4 | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}25.7}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}49.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}47.1}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}48.8}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}49.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}49.6}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}48.6}}$
VIMA-Gato |  013.3 |  013.2 |  012.2 |  012.3 |  012.8 |  013.5 |  016.8
VIMA-Flamingo |  012.3 |  011.6 |  010.7 |  012.1 |  010.7 |  011.1 |  012.1
VIMA-GPT |  011.1 |  010.3 |  012.7 |  014.2 |  011.8 |  012.1 |  012.1

###  E.3 Data Scaling

####  E.3.1 Detailed Setup

To ensure all methods are fairly pre-trained on the same amount of data (i.e.,
they have roughly the same amount of built-in information, thus the x-axis in
Fig. 4 faithfully corresponds to the extra bits of information seen during
further training), we initialize variants that directly learn from raw pixels
with MVP pre-trained ViT (Xiao et al., 2022; Radosavovic et al., 2022). It is
further MAE fine-tuned (He et al., 2021), using the _same_ in-domain data as
for the Mask R-CNN object detector. Note that the MVP pre-trained then domain
fine-tuned ViT also updates weights jointly with robot controllers later on.
We use the ViT-B backbone from MVP. The in-domain data for fine-tuning include
100 trajectories for each task.

####  E.3.2 Numerical Results

We present numerical results that constitute Fig. 4 in Table 11. The claim of
“$2.7\times$ improvement” made in Abstract and Sec. 1 is calculated as
follows. The best competing variant is VIMA-Gato that achieves $12.2\%$
average success rate trained with full data on L4. Our method trained with
$10\%$ data achieves $46\%$ average success rate on the same level. We compute
the performance gap, divide by VIMA-Gato’s performance, and only keep the
first digit after decimal to obtain the result.

Table 11: Data scaling numerical results that constitute Fig. 4. Numbers in the first row indicate the size of training dataset. Level | Method |  0000.1% |  001% |  0010% | Full (100%)
---|---|---|---|---|---
L1 | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}36.3}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}76.3}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}79.3}}$
VIMA-Gato | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  011.5 |  041.5 |  057.5
VIMA-Flamingo | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  002.0 |  037.7 |  052.3
VIMA-GPT | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  006.0 |  030.9 |  052.8
L2 | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}34.3}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}75.8}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}80.1}}$
VIMA-Gato | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  010.1 |  037.9 |  041.2
VIMA-Flamingo | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  002.0 |  033.8 |  032.6
VIMA-GPT | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  006.0 |  029.7 |  040.3
L3 | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}15.4}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}73.2}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}81.9}}$
VIMA-Gato | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  010.2 |  034.8 |  040.9
VIMA-Flamingo | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  001.0 |  033.1 |  033.6
VIMA-GPT | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  005.5 |  028.6 |  039.2
L4 | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}49.6}}$
VIMA-Gato | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  002.7 |  010.8 |  012.2
VIMA-Flamingo | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  000.5 |  011.2 |  012.0
VIMA-GPT | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  001.1 |  007.1 |  014.3

####  E.3.3 What If Baseline Variants’ ViT Is Trained from Scratch?

We further investigate what if baseline variants’ ViT is trained from scratch
and end-to-end with the robot controllers. We visualize the results in Fig.
A.21 and numerically present them in Table 12. We annotate with arrows to
indicate performance increase ($\uparrow$) and decrease ($\downarrow$). We
highlight two findings.

First, MVP pre-trained ViT is most beneficial in the setting with sufficient
in-domain training data (i.e., the 10% data scenario). It boosts the
performance for the most competing baseline variant VIMA-Gato. However, in
other settings with abundant in-domain data (i.e., the full data scenario) or
insufficient in-domain data (i.e., 1% and 0.1% scenarios), the advantage of
MVP pre-trained ViT diminishes and it even becomes detrimental. This aligns
with the finding in previous empirical studies (Hansen et al., 2022). Second,
in settings with reasonable amounts of in-domain data (i.e., the 1%, 10%, and
100% scenarios), our recommended recipe always outperforms other variants. We
notice that such a data demand generally can be satisfied by both simulated
robotics data (Mandlekar et al., 2021) and real robotics data (Dasari et al.,
2019; Brohan et al., 2022). Therefore, it demonstrates that our recommended
recipe is highly sample-efficient compared to alternative designs, especially
under practical settings.

Table 12: Data scaling when baseline variants’ ViT is trained from scratch, indicated inside parentheses. $\uparrow$ and $\downarrow$ denote performance increase and decrease. Numbers in the first row represent the size of training dataset. Level | Method |  0000.1% |  001% |  0010% | Full (100%)
---|---|---|---|---|---
L1 | Ours |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}36.3}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}76.3}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}79.3}}$
VIMA-Gato |  000.0 (${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{2.2}}\uparrow$) |  011.5 (11.9 $\uparrow$) |  041.5 (26.5 $\downarrow$) |  057.5 (57.0 $\downarrow$)
VIMA-Flamingo |  000.0 (0.0) |  002.0 (6.2 $\uparrow$)0 |  037.7 (33.9 $\downarrow$) |  052.3 (47.2 $\downarrow$)
VIMA-GPT |  000.0 (0.0) |  006.0 (17.0 $\uparrow$) |  030.9 (38.9 $\uparrow$) |  052.8 (47.9 $\downarrow$)
L2 | Ours |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}34.3}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}75.8}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}80.1}}$
VIMA-Gato |  000.0 (${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{2.0}}\uparrow$) |  010.1 (11.4 $\uparrow$) |  037.9 (24.6 $\downarrow$) |  041.2 (53.9 $\uparrow$)
VIMA-Flamingo |  000.0 (0.0) |  002.0 (6.0 $\uparrow$)0 |  033.8 (34.1 $\uparrow$) |  032.6 (47.1 $\uparrow$)
VIMA-GPT |  000.0 (0.0) |  006.0 (15.2 $\uparrow$) |  029.7 (36.7 $\uparrow$) |  040.3 (47.4 $\uparrow$)
L3 | Ours |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}15.4}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}73.2}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}81.9}}$
VIMA-Gato |  000.0 (${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{1.1}}$ $\uparrow$) |  010.2 (10.1 $\downarrow$) |  034.8 (22.6 $\downarrow$) |  040.9 (45.6 $\uparrow$)
VIMA-Flamingo |  000.0 (0.0) |  001.0 (5.4 $\uparrow$)0 |  033.1 (31.0 $\downarrow$) |  033.6 (42.1 $\uparrow$)
VIMA-GPT |  000.0 (0.0) |  005.5 (15.0 $\uparrow$) |  028.6 (35.7 $\uparrow$ ) |  039.2 (42.2 $\uparrow$)
L4 | Ours |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}49.6}}$
VIMA-Gato |  000.0 (0.0) |  002.7 (2.5 $\downarrow$) |  010.8 (5.8 $\downarrow$) |  012.2 (13.5 $\uparrow$)
VIMA-Flamingo |  000.0 (0.0) |  000.5 (0.0 $\downarrow$) |  011.2 (8.3 $\downarrow$) |  012.0 (11.1 $\downarrow$)
VIMA-GPT |  000.0 (0.0) |  001.1 (4.1 $\uparrow$) |  007.1 (9.0 $\uparrow$) |  014.3 (12.1 $\downarrow$)

Figure A.21: Data scaling when baseline variants’ ViT is trained from scratch.
In settings with reasonable amounts of in-domain data (i.e., the 1%, 10%, and
100% scenarios), our recommended recipe always outperforms other variants.

###  E.4 Vary T5 Encoder Sizes

We vary the size of the pre-trained T5 encoder (Raffel et al., 2020) to study
the effect of prompt encoding. We experiment with three T5 model capacities:
t5-small (30M), t5-base (111M), and t5-large (368M). For all T5 variants, we
fine-tune the last two layers and freeze all other layers. We fix the
parameter count of the decision-making part to be 200M. As shown in Table 13,
we find no significant difference among the variants. Thus we set the standard
t5-base as default for all our models.

Table 13: Performances of our method with differently sized pre-trained T5 prompt encoder. We fix the parameter count of the decision-making part to be 200M. | t5-small (30M) | t5-base (111M) | t5-large (368M)
---|---|---|---
L1 | 78.8 | 81.5 | 80.8
L2 | 79.0 | 81.5 | 81.0
L3 | 80.3 | 78.7 | 81.0
L4 | 49.1 | 48.6 | 49.3

###  E.5 Policy Robustness

##### Increasing Amounts of Distractors.

We study the policy robustness against increasing amounts of distractors in
scenes. For all tasks being evaluated, we add one more distractor object. We
run our largest VIMA model with 200M parameters. The result is presented in
Table 14.

It turns out that the performance of VIMA degrades minimally with more
distractors than the training distribution. This indicates that our agent has
learned a reasonably robust policy against objects that are irrelevant to the
task.

Table 14: Evaluation results on tasks with increased amounts of distractors. We fix the parameter count of the decision-making part to be 200M. | L1 | L2 | L3 | L4
---|---|---|---|---
Original | 81.5 | 81.5 | 78.7 | 48.6
More Distractors | 78.5 | 78.6 | 72.9 | 47.8
Relevant Performance Decrease (%) |  03.6 |  03.5 |  07.3 |  01.6

##### Imperfect Prompts.

We then study the policy robustness against imperfect prompts, including
incomplete prompts (randomly masking out words with <UNK> token) and corrupted
prompts (randomly swapping words, which could have changed the task meaning
altogether). We run our largest VIMA model with 200M parameters, results are
shown in Table 15.

Our well-trained model exhibits minimal performance decrease when evaluated on
masked prompts and minor decrease on corrupted prompts. We attribute this
robustness to the high-quality pre-trained T5 language backbone.

Table 15: Evaluation results with incomplete and corrupted prompts. We fix the parameter count of the decision-making part to be 200M. | L1 | L2 | L3 | L4
---|---|---|---|---
Original | 81.5 | 81.5 | 78.7 | 48.6
Incomplete Prompts | 80.8 | 81.1 | 77.0 | 48.0
Corrupted Prompts | 78.2 | 78.1 | 73.8 | 45.3
Relevant Performance Decrease w/ Incomplete Prompts (%) |  00.8 |  00.4 |  02.1 |  01.2
Relevant Performance Decrease w/ Corrupted Prompts (%) |  04.2 |  04.3 |  06.6 |  07.2

##  Appendix F Extended Related Work

In this section, we provide an extended review of related work as
complementary to Section 6.

##### Multi-Task Learning by Sequence Modeling.

In computer vision, Mask R-CNN (He et al., 2017), UberNet (Kokkinos, 2016),
and 12-in-1 (Lu et al., 2020) leverage a single backbone model with multiple
independent heads for different tasks. UVim (Kolesnikov et al., 2022) is
another unified approach for vision that uses a language model to generate the
guiding code for a second model to predict raw vision outputs. In multimodal
learning, numerous works (Lu et al., 2022; Wang et al., 2022a; Zellers et al.,
2021, 2022; Buch et al., 2022; Fu et al., 2021; Yang et al., 2022) investigate
the unification of image, video, audio, and/or language modalities to deliver
multi-purpose foundation models, although most of which are not equipped with
decision-making capabilities. BEiT-3 (Wang et al., 2022c) performs masked data
modeling on images, texts and image-text pairs to pre-train a backbone for
various downstream tasks. MetaMorph (Gupta et al., 2022a) learns a universal
controller over a modular robot design space.

##### Foundation Models for Embodied Agents.

Embodied agent research (Duan et al., 2022; Batra et al., 2020; Ravichandar et
al., 2020; Collins et al., 2021) is adopting the large-scale pre-training
paradigm (Yang et al., 2023), powered by a collection of learning environments
(Abramson et al., 2020; Shridhar et al., 2020; Savva et al., 2019; Puig et
al., 2018; Team et al., 2021; Toyama et al., 2021; Shi et al., 2017). From the
aspect of pre-training for better representations, Reid et al. (2022) fine-
tunes from LLM checkpoints to accelerate policy learning. LaTTe (Bucker et
al., 2022) and Embodied-CLIP (Khandelwal et al., 2021) leverage the frozen
visual and textual representations of CLIP (Radford et al., 2021) for robotic
manipulation. MaskDP (Liu et al., 2022a) pre-trains bidirectional transformers
for various downstream embodied tasks. From the perspective of leveraging
transformer as agent architecture, methods such as Dasari & Gupta (2020) and
MOSAIC (Zhao et al., 2022) achieve superior performance in one-shot video
imitation tasks. They both use the self-attention mechanism with auxiliary
losses such as inverse dynamics loss (Dasari & Gupta, 2020) and contrastive
loss (Zhao et al., 2022) to learn robot controllers. InstructRL (Liu et al.,
2022b) leverages jointly pre-trained vision-language models as robot agents to
perform manipulation tasks. From the perspective of large language models for
robot learning, Socratic Models (Zeng et al., 2022) composes multiple vision
and language foundation models for multimodal reasoning in videos. ROSIE (Yu
et al., 2023) leverages text-to-image diffusion models to augment existing
robotic dataset (Brohan et al., 2022) via inpainting. MOO (Minderer et al.,
2022) adopts a similar object-centric representation as ours for open-world
object manipulation. Furthermore, Voyager (Wang et al., 2023) develops a LLM-
powered agent operating in an open-ended virtual world (Fan et al., 2022).

##### Robot Manipulation and Benchmarks.

There are many prior works that are not mentioned in the main paper that study
different robotic manipulation tasks, such as instruction following (Shridhar
et al., 2021; Lynch & Sermanet, 2021), constraint satisfaction (Bharadhwaj et
al., 2021; Srinivasan et al., 2020; Thananjeyan et al., 2021), one-shot
imitation (Paine et al., 2018; Huang et al., 2019; Dasari & Gupta, 2020;
Aceituno et al., 2021; Zhao et al., 2022), rearrangement (Weihs et al., 2021;
Szot et al., 2021; Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021;
Stengel-Eskin et al., 2022), and reasoning (Gupta et al., 2019; Ahmed et al.,
2021; Toyer et al., 2020; Lim et al., 2021). Multiple simulation benchmarks
are introduced to study the above tasks: 1) Indoor simulation environments:
Habitat (Savva et al., 2019; Szot et al., 2021) is equipped with a high-
performance 3D simulator for fast rendering and proposes a suite of common
tasks for assistive robots. 2) Tabletop environments: RLBench (James et al.,
2019) and SURREAL (Fan et al., 2018, 2019) are other widely used simulator
benchmarks studying robotics manipulation with tabletop settings. STRETCH-P&P
(Zhang & Weihs, 2023) studies generalization across goals for reset-free
reinforcement learning. All these aforementioned simulators and benchmarks do
not natively support task specification and prompting with multiple
modalities.

##  Appendix G Limitations and Further Discussions

##### Reliance on a separate object detector.

VIMA inherits the errors from the standalone object detector, which may
struggle in the cases of occlusion or out-of-distribution object forms.
However, using object detectors is not entirely without merits. First, it
allows us to seamlessly switch to stronger detection models when they become
available. For example, we can switch to object detectors that are more robust
and open-vocabulary, such as OWL-ViT (Minderer et al., 2022). This would
enable VIMA to transfer to real-world scenarios with minimal modifications.
Second, by leveraging pre-trained vision pipelines, several concurrent works
have demonstrated the superiority of object-centric representation in robot
manipulation. For example, VIOLA (Zhu et al., 2022) achieves better
performance with a pre-trained Region Proposal Network (Ren et al., 2015). MOO
(Stone et al., 2023) also shows that a robot agent with OWL-ViT (Minderer et
al., 2022) as the object detector significantly outperforms RT-1 (Brohan et
al., 2022), which directly learns from raw pixels, on various real-world
manipulation tasks. In fact, MOO (Stone et al., 2023) includes a baseline
called “VIMA-like” that already demonstrates strong performance on real robots
under real-world scenarios. As we witness image segmentation is becoming more
robust and general-purpose (Kirillov et al., 2023), we envision such design
choice will become more effective and further gain more popularity.

##### Limited simulator realism and task complexity.

Our goal with VIMA-Bench is to explore the multi-task ability, generalization,
and understanding of multi-modality. Therefore, these aspects are not the
primary focus of this work. However, we envision future works can combine this
formulation with more physically realistic simulators such as Zhu et al.
(2020), Srivastava et al. (2021), and Mittal et al. (2023).

##### Limited action primitives.

We inherit the same high-level action space from well-established prior works,
such as Transporter (Zeng et al., 2020). While “pick-and-place” and “wipe”
seem simple, they do cover a wide range of tabletop manipulation tasks and are
crucial to industrial use cases like warehouse robots (Yoon et al., 2003;
Berscheid et al., 2020; Devin et al., 2020; Song et al., 2019). While VIMA is
currently using these two actions, the algorithm design is general-purpose and
does not make assumptions about the particular action choices. For example,
VIMA would require only minimal modifications to support more low-level action
spaces like joint-torque control.

##  Appendix H Full Tables

This section contains more detailed tables that correspond to the results in
Figure 4. Specifically, we show breakdown results on each task that constitute
the model scaling results in Tables 16, 17, 18, and 19.

Table 16: L1 level generalization results. Model indicates robot controller
parameter count. Integers in the first row refer to indices of tasks described
in Appendix, Sec. B.

Model | Method | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 09 | 11 | 12 | 15 | 16 | 17
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
2M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.0}}$ |  037.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}9.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}87.0}}$ |  064.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}93.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}45.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}63.0}}$
VIMA-Gato |  062.0 |  061.0 |  022.5 |  013.5 |  007.0 |  044.5 |  054.0 |  004.0 |  048.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}85.0}}$ |  044.5 |  043.0 |  000.0
VIMA-Flamingo |  056.0 |  056.0 |  053.5 |  036.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}37.5}}$ |  045.0 |  055.5 |  003.5 |  054.0 |  083.5 |  040.5 |  028.5 |  002.0
VIMA-GPT |  059.5 |  050.5 |  007.5 |  007.0 |  000.5 |  043.5 |  049.5 |  002.0 |  061.5 |  076.5 |  027.5 |  005.0 |  000.0
4M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}45.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}10.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}90.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}90.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}51.0}}$
VIMA-Gato |  061.0 |  061.5 |  008.0 |  046.0 |  032.5 |  045.5 |  057.0 |  001.0 |  064.5 |  086.0 |  046.5 |  042.5 |  002.0
VIMA-Flamingo |  061.0 |  062.0 |  057.5 |  047.5 |  045.0 |  049.5 |  059.5 |  005.5 |  080.0 |  083.5 |  040.5 |  043.0 |  002.0
VIMA-GPT |  058.0 |  055.0 |  017.5 |  025.0 |  012.0 |  047.5 |  054.5 |  003.0 |  059.5 |  080.5 |  027.0 |  041.5 |  000.5
9M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}51.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}13.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}82.5}}$ |  058.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}42.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}63.5}}$
VIMA-Gato |  059.0 |  061.0 |  041.0 |  050.5 |  038.5 |  047.5 |  059.5 |  009.5 |  058.0 |  080.5 |  044.0 |  024.0 |  002.5
VIMA-Flamingo |  058.5 |  060.0 |  046.0 |  049.0 |  042.5 |  045.5 |  060.5 |  004.0 |  066.5 |  081.5 |  036.5 |  041.5 |  001.0
VIMA-GPT |  058.5 |  054.5 |  040.5 |  047.5 |  037.5 |  047.5 |  058.5 |  009.0 |  072.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}85.0}}$ |  038.5 |  034.0 |  001.0
20M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}59.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}13.5}}$ |  074.0 |  072.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.5}}$ |  039.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}47.5}}$
VIMA-Gato |  061.5 |  062.0 |  032.5 |  049.0 |  038.0 |  046.0 |  060.0 |  005.0 |  068.0 |  083.0 |  047.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.5}}$ |  002.0
VIMA-Flamingo |  063.0 |  061.5 |  055.0 |  050.0 |  042.5 |  041.5 |  058.0 |  006.0 |  062.0 |  083.0 |  044.0 |  038.5 |  001.0
VIMA-GPT |  060.5 |  064.0 |  050.5 |  044.0 |  041.0 |  048.0 |  061.5 |  007.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}85.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}84.0}}$ |  044.5 |  039.0 |  002.5
43M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}57.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}15.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}86.0}}$ |  069.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ |  040.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}51.5}}$
VIMA-Gato |  057.0 |  065.5 |  059.0 |  057.5 |  043.5 |  050.0 |  056.0 |  005.0 |  067.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}83.5}}$ |  063.0 |  037.0 |  000.0
VIMA-Flamingo |  054.5 |  057.0 |  054.5 |  054.0 |  045.0 |  043.5 |  055.5 |  006.0 |  067.5 |  082.5 |  049.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}40.5}}$ |  001.5
VIMA-GPT |  058.0 |  060.5 |  069.5 |  053.5 |  041.5 |  047.0 |  055.5 |  004.0 |  066.5 |  081.5 |  045.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}40.5}}$ |  001.5
92M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}58.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}14.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}80.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}92.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$ |  040.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}48.5}}$
VIMA-Gato |  076.5 |  059.5 |  090.0 |  056.5 |  044.5 |  048.5 |  068.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}14.0}}$ |  064.5 |  089.5 |  085.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}43.0}}$ |  001.5
VIMA-Flamingo |  056.0 |  056.0 |  065.5 |  050.5 |  041.0 |  048.0 |  056.0 |  003.0 |  070.0 |  087.0 |  041.5 |  038.0 |  002.0
VIMA-GPT |  057.0 |  057.5 |  058.5 |  053.0 |  045.0 |  051.0 |  061.0 |  008.0 |  065.5 |  087.0 |  046.0 |  033.0 |  001.0
200M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}56.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}18.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}77.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}93.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}76.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}43.0}}$
VIMA-Gato |  079.0 |  068.0 |  091.5 |  057.0 |  044.5 |  054.0 |  074.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}18.0}}$ |  061.0 |  088.5 |  083.5 |  033.5 |  002.5
VIMA-Flamingo |  056.0 |  058.5 |  063.0 |  048.5 |  038.0 |  048.5 |  062.5 |  003.5 |  066.5 |  086.0 |  040.0 |  043.5 |  002.5
VIMA-GPT |  062.0 |  057.5 |  041.0 |  055.5 |  045.5 |  047.5 |  054.5 |  008.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}77.0}}$ |  081.5 |  041.0 |  038.0 |  000.5

Table 17: L2 level generalization results. Model indicates robot controller
parameter count. Integers in the first row refer to indices of tasks described
in Appendix, Sec. B.

Model | Method | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 09 | 11 | 12 | 15 | 16 | 17
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
2M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}95.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}37.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}87.5}}$ |  067.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}54.5}}$
VIMA-Gato |  049.5 |  049.0 |  023.0 |  017.5 |  005.0 |  047.5 |  046.5 |  005.5 |  050.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}82.5}}$ |  049.0 |  042.0 |  000.5
VIMA-Flamingo |  045.5 |  046.0 |  056.0 |  039.5 |  035.5 |  049.0 |  047.0 |  009.0 |  053.0 |  080.0 |  043.0 |  029.5 |  001.0
VIMA-GPT |  051.0 |  045.5 |  009.5 |  007.0 |  000.5 |  045.5 |  045.0 |  000.0 |  065.0 |  081.5 |  032.0 |  005.0 |  000.0
4M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}44.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}14.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}89.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}91.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}95.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}43.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}52.5}}$
VIMA-Gato |  044.5 |  052.0 |  009.0 |  039.0 |  028.0 |  049.5 |  048.5 |  002.0 |  064.0 |  086.5 |  044.5 |  042.5 |  002.0
VIMA-Flamingo |  049.5 |  050.5 |  051.0 |  048.0 |  043.0 |  050.5 |  053.5 |  005.5 |  081.5 |  082.5 |  048.5 |  039.5 |  001.0
VIMA-GPT |  050.5 |  049.5 |  016.5 |  025.5 |  012.0 |  041.0 |  047.0 |  004.0 |  063.0 |  079.0 |  028.5 |  039.0 |  000.0
9M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}49.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}19.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}80.5}}$ |  065.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}95.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}42.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}66.0}}$
VIMA-Gato |  047.0 |  044.5 |  039.5 |  046.5 |  037.5 |  048.5 |  051.0 |  005.5 |  059.0 |  083.0 |  051.5 |  023.5 |  001.0
VIMA-Flamingo |  048.0 |  047.5 |  049.0 |  052.5 |  042.0 |  047.5 |  048.5 |  008.5 |  066.0 |  081.5 |  045.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}42.0}}$ |  002.0
VIMA-GPT |  048.5 |  047.0 |  043.5 |  047.0 |  037.0 |  047.5 |  045.5 |  010.5 |  074.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}85.0}}$ |  043.5 |  033.0 |  001.0
20M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}61.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}16.5}}$ |  075.5 |  075.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.0}}$ |  037.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}47.5}}$
VIMA-Gato |  044.0 |  051.5 |  039.0 |  051.0 |  038.5 |  047.5 |  052.5 |  006.0 |  065.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}84.0}}$ |  052.5 |  040.5 |  001.0
VIMA-Flamingo |  048.5 |  049.0 |  055.5 |  048.0 |  042.5 |  046.5 |  052.0 |  006.0 |  066.0 |  082.0 |  047.5 |  037.0 |  000.5
VIMA-GPT |  050.5 |  049.5 |  053.0 |  044.5 |  043.5 |  047.0 |  046.0 |  008.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}83.5}}$ |  080.0 |  046.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}41.0}}$ |  002.5
43M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}54.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}14.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}83.5}}$ |  069.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.0}}$ |  038.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}51.5}}$
VIMA-Gato |  050.0 |  051.5 |  053.0 |  057.5 |  042.5 |  047.0 |  051.0 |  008.5 |  067.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}83.0}}$ |  063.5 |  032.0 |  000.5
VIMA-Flamingo |  048.0 |  046.5 |  052.0 |  051.5 |  043.5 |  045.0 |  051.5 |  005.0 |  068.0 |  081.5 |  052.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}44.0}}$ |  001.5
VIMA-GPT |  045.0 |  049.0 |  064.5 |  053.5 |  040.0 |  046.5 |  048.5 |  008.5 |  068.0 |  082.0 |  050.0 |  040.0 |  001.5
92M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}57.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}19.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}81.5}}$ |  092.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}42.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}53.5}}$
VIMA-Gato |  064.5 |  050.0 |  083.0 |  056.5 |  046.0 |  055.5 |  054.5 |  010.5 |  064.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}92.5}}$ |  081.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}42.0}}$ |  001.0
VIMA-Flamingo |  053.0 |  048.5 |  067.5 |  053.0 |  043.0 |  049.0 |  053.0 |  004.5 |  067.0 |  084.0 |  050.0 |  040.0 |  001.0
VIMA-GPT |  050.5 |  055.0 |  055.5 |  054.5 |  043.0 |  051.5 |  054.5 |  010.5 |  068.5 |  087.0 |  049.5 |  034.0 |  003.0
200M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}54.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}77.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}93.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}75.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}45.0}}$
VIMA-Gato |  056.5 |  053.5 |  088.0 |  055.5 |  043.5 |  055.5 |  053.0 |  014.0 |  063.0 |  090.5 |  081.5 |  033.0 |  004.0
VIMA-Flamingo |  051.0 |  052.5 |  061.5 |  049.5 |  038.5 |  047.5 |  055.5 |  005.5 |  070.5 |  082.0 |  042.0 |  039.0 |  003.0
VIMA-GPT |  052.0 |  052.0 |  049.5 |  054.5 |  045.5 |  052.5 |  051.0 |  011.0 |  076.5 |  084.0 |  043.0 |  038.0 |  000.5

Table 18: L3 level generalization results. Model indicates robot controller
parameter count. Integers in the first row refer to indices of tasks described
in Appendix, Sec. B.

Model | Method | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 09 | 11 | 15 | 16 | 17
---|---|---|---|---|---|---|---|---|---|---|---|---|---
2M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}34.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}94.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}48.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}39.0}}$
VIMA-Gato |  045.5 |  048.0 |  028.0 |  023.0 |  003.0 |  045.5 |  045.0 |  002.5 |  040.5 |  029.5 |  037.0 |  001.0
VIMA-Flamingo |  041.5 |  054.5 |  050.5 |  039.5 |  029.0 |  045.0 |  049.5 |  005.5 |  057.5 |  022.5 |  025.0 |  000.0
VIMA-GPT |  048.5 |  050.0 |  005.0 |  007.0 |  002.5 |  047.0 |  045.5 |  002.0 |  069.5 |  022.5 |  005.0 |  000.0
4M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}44.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}12.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}92.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}47.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}43.5}}$
VIMA-Gato |  044.5 |  055.0 |  009.5 |  037.5 |  024.5 |  047.0 |  050.0 |  003.5 |  060.0 |  030.5 |  037.5 |  000.0
VIMA-Flamingo |  046.0 |  053.5 |  059.0 |  049.5 |  035.5 |  047.5 |  048.0 |  007.0 |  087.5 |  030.5 |  039.5 |  000.0
VIMA-GPT |  044.0 |  047.0 |  014.5 |  022.0 |  009.0 |  039.5 |  040.0 |  002.0 |  062.0 |  028.5 |  043.0 |  001.0
9M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}44.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}18.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}88.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}48.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.5}}$
VIMA-Gato |  044.5 |  053.5 |  042.5 |  052.0 |  028.0 |  046.5 |  051.5 |  006.0 |  067.0 |  035.0 |  023.0 |  000.5
VIMA-Flamingo |  044.5 |  053.0 |  053.0 |  048.5 |  033.0 |  041.0 |  045.5 |  008.0 |  072.5 |  027.0 |  044.5 |  000.5
VIMA-GPT |  049.0 |  050.5 |  039.0 |  046.5 |  030.5 |  043.0 |  052.0 |  006.5 |  084.0 |  031.5 |  035.0 |  000.5
20M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}55.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}15.0}}$ |  088.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}44.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}29.5}}$
VIMA-Gato |  046.5 |  055.0 |  044.5 |  057.0 |  031.5 |  047.5 |  051.5 |  002.5 |  072.5 |  030.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}44.0}}$ |  000.0
VIMA-Flamingo |  047.0 |  054.5 |  053.0 |  055.0 |  036.0 |  042.5 |  048.0 |  006.5 |  070.0 |  033.0 |  041.5 |  000.0
VIMA-GPT |  050.0 |  060.5 |  056.5 |  048.0 |  033.5 |  051.0 |  046.0 |  006.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}92.5}}$ |  032.5 |  043.5 |  001.5
43M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}47.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}18.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}93.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}45.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}84.0}}$
VIMA-Gato |  044.0 |  055.0 |  059.5 |  058.0 |  034.0 |  049.0 |  054.0 |  007.0 |  074.0 |  040.0 |  035.0 |  000.5
VIMA-Flamingo |  047.0 |  054.0 |  056.5 |  052.5 |  037.0 |  046.5 |  044.5 |  006.5 |  069.5 |  027.0 |  043.0 |  000.0
VIMA-GPT |  047.5 |  057.0 |  061.0 |  050.0 |  034.5 |  048.0 |  053.5 |  008.0 |  074.0 |  040.5 |  041.5 |  000.5
92M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}58.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ |  013.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}94.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ |  042.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}82.5}}$
VIMA-Gato |  061.5 |  054.0 |  073.0 |  056.0 |  036.0 |  050.0 |  048.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.0}}$ |  066.5 |  044.0 |  041.5 |  000.0
VIMA-Flamingo |  051.0 |  051.5 |  068.0 |  051.5 |  036.5 |  050.5 |  047.0 |  006.0 |  069.5 |  028.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}45.5}}$ |  000.5
VIMA-GPT |  050.0 |  056.5 |  063.0 |  052.5 |  032.0 |  049.5 |  053.0 |  005.0 |  078.0 |  034.5 |  037.5 |  000.0
200M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}54.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}17.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}90.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}46.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}43.5}}$
VIMA-Gato |  051.0 |  058.0 |  084.5 |  056.5 |  035.5 |  053.5 |  049.0 |  015.0 |  065.0 |  052.0 |  033.0 |  000.0
VIMA-Flamingo |  049.0 |  050.0 |  066.5 |  047.0 |  035.0 |  047.5 |  050.0 |  004.0 |  066.0 |  030.5 |  043.5 |  000.5
VIMA-GPT |  052.0 |  051.0 |  055.0 |  049.5 |  040.0 |  046.0 |  050.5 |  005.0 |  082.0 |  037.0 |  038.0 |  001.5

Table 19: L4 level generalization results. Model indicates robot controller parameter count. Integers in the first row refer to indices of tasks described in Appendix, Sec. B. Model | Method | 08 | 10 | 13 | 14
---|---|---|---|---|---
2M | Ours |  006.5 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.5}}$
VIMA-Gato |  021.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  032.0
VIMA-Flamingo |  022.0 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  027.5
VIMA-GPT | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}22.5}}$ |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  022.0
4M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.0}}$ |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$
VIMA-Gato |  017.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}2.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  034.0
VIMA-Flamingo |  017.0 |  000.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  029.0
VIMA-GPT |  019.0 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  022.5
9M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}92.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}96.5}}$
VIMA-Gato |  018.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  031.0
VIMA-Flamingo |  021.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  021.5
VIMA-GPT |  020.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  030.5
20M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}95.5}}$
VIMA-Gato |  020.5 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  029.0
VIMA-Flamingo |  021.0 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  027.5
VIMA-GPT |  020.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  036.0
43M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}99.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}97.0}}$
VIMA-Gato |  021.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  030.5
VIMA-Flamingo |  018.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  024.5
VIMA-GPT |  017.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  030.0
92M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}98.5}}$
VIMA-Gato |  022.0 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  032.0
VIMA-Flamingo |  019.5 |  000.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  025.0
VIMA-GPT |  018.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.5}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  029.5
200M | Ours | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{100.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{0}94.5}}$
VIMA-Gato |  030.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  037.0
VIMA-Flamingo |  024.5 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  024.0
VIMA-GPT |  020.0 | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ | ${\color[rgb]{0.09,0.45,0.27}\definecolor[named]{pgfstrokecolor}{rgb}{0.09,0.45,0.27}\mathbf{\hphantom{00}0.0}}$ |  028.5