ManiSkill Evaluation#

ManiSkill is a SAPIEN-based robot manipulation benchmark. The LoongForge eval module’s default task is PickCube-v1 with pd_ee_delta_pose control (7D action).

Current status: no released ManiSkill-domain weights for pi05/xvla. Shipped configs run as connectivity checks (server.random_init: true), not scores.

Note: xvla cannot run with real ManiSkill weights yet — it emits absolute EE poses while ManiSkill only ships the delta controller (pd_ee_delta_pose); the required absolute-EE control mode is not implemented, so the runner raises NotImplementedError for a non-random_init xvla run (see the model integration guide §4.2 Control mode).

Step 0: Download weights#

Model	Weights
pi05	none released (connectivity only)
xvla	none released (connectivity only)

Step 1: Environment setup#

Standard environment#

Install ManiSkill following the official repository instructions (SAPIEN-based), then install the additional dependencies:

pip install websockets msgpack pyyaml

⚠️ Common issues:

Vulkan / SAPIEN. Verify a real NVIDIA Vulkan ICD with vulkaninfo (expect deviceName = NVIDIA ...); set LD_LIBRARY_PATH / VK_ICD_FILENAMES before ManiSkill/SAPIEN is imported.

Step 2: Run evaluation#

Run from inside the benchmark environment. The run scripts and eval YAMLs ship with /path/to/... placeholders — fill them in before running:

cd /path/to/LoongForge-VLA
examples/embodied/pi05/eval/run_maniskill_eval.sh    # pi05 (connectivity only)
examples/embodied/xvla/eval/run_maniskill_eval.sh    # xvla (connectivity only)

Environment variables: CONFIG, BENCHMARK_PYTHON (ManiSkill env interpreter), CUDA_VISIBLE_DEVICES, plus the SAPIEN Vulkan variables.

Key config fields (see examples/embodied/<model>/eval/configs/maniskill/pick_cube_smoke.yaml):

benchmark.task_name — PickCube-v1 (default)
benchmark.control_mode — pd_ee_delta_pose (7D action)
server.random_init — true in shipped configs. With ManiSkill-domain weights set false, fill ckpt_path / dataset_statistics_path (state dim is typically 8: 7 arm joints + gripper width)

Verification#

Model	Status	Notes
pi05	connectivity only	needs ManiSkill-domain weights + `dataset_statistics.json`
xvla	connectivity only	needs ManiSkill-domain weights + absolute-EE control mode (not implemented, see above)

ManiSkill Evaluation

Contents