VLABench/pi05-primitive-10task

Pi05 official implementation trained on VLABench datasets.

This repository provides the official release of the Pi 0.5 model with the stop-gradient mechanism enabled, trained with the whole VLABench's official primitive tasks dataset.

Evaluation

To run this checkpoint, please clone this repo: https://github.com/Shiduo-zh/openpi, and checkout to the branch pi05. Assume that you download this checkpoints and put it in the directory checkpoints, to run the policy as server, please run:

bash vla_bench_scipts/serve_policy.sh pi05_ft_vlabench_primitive checkpoints/VLABench/pi05-primitive-10task/sg_pi05_base/29999/

After serving the policy, open another terminal and run:

bash vla_bench_scipts/multi_run_vlabench.sh <Your path to store the evaluate results>

Train

To reproduce the training result, please run the training script with the config pi05_ft_vlabench_primitive.

XLA_PYTHON_CLIENT_MEM_FRACTION=0.95 uv run scripts/train.py pi05_ft_vlabench_primitive --exp-name=pi05_ft_vlabench_primitive --overwrite

Our checkpoint is trained on 8 H100 for 30k iterations, with 5000 episodes data acrossing 10 tasks.

Reference Results

The reference success rate of this model is:

Track	add_condiment	insert_flower	select_book	select_chemistry_tube	select_drink	select_fruit	select_mahjong	select_painting	select_poker	select_toy	Avg_SR
track_1_in_distribution	0.56	0.22	0.545	0.367	0.44	0.42	0.38	0.3	0.52	0.3	0.406
track_2_cross_category	0.06	0.04	0.02	0.16	0.10	0.36	0.27	0.32	0.5	0.24	0.226
track_3_common_sense	0.2	0.1	0.383	0.32	0.1	0.2	0	0.24	0.2	0.06	0.18
track_4_semantic_instruction	0.02	0.1	0.333	0.245	0.30	0.14	0.205	0.12	0.04	0.074	0.161
track_6_unseen_texture	0.48	0.1	0.383	0.354	0.18	0.2	0.217	0.3	0.32	0.08	0.256