Benchmark Environments#
This document records the benchmark runtime environments used by the LoongForge-VLA eval module under loongforge/embodied/eval.
For installation, follow the official benchmark homepages — each benchmark page links to its official repository. This document is not an install guide; it records the dependency versions of the environments currently used by the eval module, so they can be compared / reproduced when debugging.
Environment isolation. The benchmark client and the policy server run as separate processes with different dependencies (see the module overview Architecture section). We use conda environments for this isolation; other approaches — e.g. running the two sides in separate pods — work too, but then you are responsible for the inter-process communication yourself (the two sides must reach each other over the configured host/port).
The version lists below are the verified combinations of the internal envs.
Tool Version#
conda 26.3.2
LIBERO#
Runtime Python:
/path/to/conda/envs/libero/bin/python
Selected dependency versions:
python 3.8.13
torch 2.1.2
numpy 1.24.4
libero 0.1.0 dev
robosuite 1.4.0
mujoco 3.2.3
gym 0.25.2
imageio 2.35.1
imageio-ffmpeg 0.5.1
opencv-python 4.6.0.66
websockets 13.1
msgpack 1.1.1
pyyaml 6.0.3
Used by:
examples/embodied/pi05/eval/configs/libero/*.yaml
CALVIN#
Runtime Python:
/path/to/conda/envs/calvin/bin/python
CALVIN evaluation uses the original-format CALVIN validation dataset and config assets, including validation/.hydra/merged_config.yaml, calvin_models/conf, and eval_sequences.json. LeRobot-format CALVIN datasets are useful for training/statistics but are not sufficient by themselves for official online long-horizon rollout.
Current CALVIN status:
benchmark env: /path/to/conda/envs/calvin
repo/config assets: /path/to/calvin
validation dataset: /path/to/calvin_debug_dataset
pi05 configs: only smoke.yaml
server.random_init: true (connectivity only, not task success)
connectivity status: pass; 1 sequence, first subtask capped at 30 steps
model score status: need CALVIN-domain weights + dataset_statistics.json
Used by:
examples/embodied/pi05/eval/configs/calvin/smoke.yaml
SimplerEnv#
Runtime Python:
/path/to/conda/envs/simplerenv/bin/python
Selected dependency versions:
python 3.10.20
numpy 1.24.4
mani-skill2-real2sim 0.5.3
sapien 2.2.2
gymnasium 0.29.1
imageio 2.37.3
imageio-ffmpeg 0.6.0
opencv-python 4.13.0.92
websockets 16.0
msgpack 1.1.2
pyyaml 6.0.3
The SimplerEnv runner prepares LD_LIBRARY_PATH, VK_ICD_FILENAMES, and XDG_RUNTIME_DIR (SAPIEN Vulkan requirements, see the SimplerEnv page ⚠️ common issues), then re-execs the benchmark Python process once before constructing the environment.
Current SimplerEnv status:
X-VLA WidowX status: task success after absolute EE controller patch
(see patches/simplerenv/xvla.md)
configs: examples/embodied/xvla/eval/configs/simplerenv/*
GR00T-N1.6 status: task success (open_drawer 10/10, close_drawer 9/10,
spoon 7/10, eggplant 6/10, carrot 16/30, stack_cube 2/10;
official prepackaged_config)
uses the stock upstream delta controller, no env change
configs: examples/embodied/groot_n1_6/eval/configs/simplerenv/*
pi05 configs: only widowx_stack_cube_smoke.yaml
server.random_init: true (connectivity only, not task success)
other Bridge tasks: edit task_name in-file comments
Upstream SimplerEnv without the 255isWhite-style absolute EE registration will mis-execute absolute pose actions as deltas. Prefer cloning the fork documented in patches/simplerenv/xvla.md or applying the two manual patches.
Used by:
examples/embodied/pi05/eval/configs/simplerenv/*.yaml
examples/embodied/xvla/eval/configs/simplerenv/*.yaml
RoboTwin#
Runtime Python:
/path/to/conda/envs/robotwin/bin/python
Selected dependency versions:
python 3.10.20
torch 2.4.1
numpy 1.26.4
sapien 3.0.0b1
imageio 2.34.2
imageio-ffmpeg 0.6.0
opencv-python 4.11.0.86
websockets 16.0
msgpack 1.1.2
msgpack-numpy 0.4.8
pyyaml 6.0.3
Video logging dependency:
/path/to/conda/envs/robotwin/bin/ffmpeg
ffmpeg version 7.0.2-static
The ffmpeg executable is provided by the installed imageio-ffmpeg package and linked into the robotwin env bin directory so RoboTwin official video logging can launch ffmpeg directly.
Current RoboTwin status (2026-07-21):
Official evaluator: script/eval_policy.py via robotwin_runner + bridges/robotwin_policy.py
action_bridge modes: pi05_aloha_14d | ee6d_dual
pi05 RoboTwin2: task success (adjust_bottle demo_clean)
action_bridge=pi05_aloha_14d, action_dim=14, action_horizon=32
weight example: /path/to/pi0.5_robotwin2
stats: examples/embodied/pi05/eval/assets/pi05_robotwin2_dataset_stats.json
(from the weight's assets/.../norm_stats.json: state→observation.state, actions→action)
xvla RoboTwin2: task success (adjust_bottle demo_clean)
action_bridge=ee6d_dual, domain_id=6
weight example: /path/to/X-VLA-RoboTwin2
Connectivity only: edit adjust_bottle_smoke*.yaml (random_init); no separate YAML
Used by:
examples/embodied/pi05/eval/configs/robotwin/*.yaml
examples/embodied/xvla/eval/configs/robotwin/*.yaml
ManiSkill#
Runtime Python:
/path/to/conda/envs/maniskill/bin/python
Current ManiSkill status:
pi05 configs: only pick_cube_smoke.yaml
server.random_init: true (connectivity only, not task success)
Task: PickCube-v1, 7D, pd_ee_delta_pose
Model score status: need ManiSkill-domain weights + dataset_statistics.json
Visual smoke needs the SAPIEN Vulkan runtime (see the SimplerEnv page ⚠️ common issues). The ManiSkill runner prepares the NVIDIA ICD and library path before importing ManiSkill/SAPIEN.
Used by:
examples/embodied/pi05/eval/configs/maniskill/*.yaml