Guide to Integrating a New Model into the Eval System#

This document summarizes the complete workflow, key configuration items, and the predict_action interface contract that the model side must implement in order to integrate a new VLA model into the loongforge/embodied/eval eval system. Beyond implementing a Factory + PayloadBuilder (+ an optional ActionDecoder) and writing the YAML config, integration also involves a series of model-semantics-level configuration points that must be confirmed one by one. pi05 and xvla differ almost entirely on these configuration points, so this document uses the two as examples to walk through each item and provides a configuration comparison table.

Scope note: The semantic checklist below covers all benchmarks currently integrated (LIBERO / RoboTwin / SimplerEnv / CALVIN / ManiSkill). Early integration used LIBERO as the primary acceptance path, so some items are illustrated with LIBERO; cross-benchmark protocol differences (such as RoboTwin action_bridge) are called out separately. For the config layout conventions see §1 Overview; for task success status see ../eval_overview.md / user_guide.md.

Overall architecture (a four-stage chain that decouples the model from the benchmark):

Adapter (obs -> canonical, exposes state_raw)
  -> PayloadBuilder (canonical -> predict_action kwargs, per model, capabilities declared via class attributes)
  -> model.predict_action (policy server, server side)
  -> ActionDecoder (raw chunk -> env action; key auto-assembled as {action_encoding}_to_{action_space}, identity -> IdentityDecoder)

1. Overview#

Integrating a new model = Factory + PayloadBuilder + optional ActionDecoder, with the three separated by responsibility (component details in §3 Component details). Add new files, and do not modify existing files.

Components — responsibilities:

ModelFactory (server side, requires torch) — factories/<model>_factory.py, registered via @register_factory("<model_type>"); build() returns PredictActionModelSpec(model, metadata). Responsibilities:
- load the model: import, config, weight loading / random_init, device/dtype
- wrap it behind the unified predict_action(images, instructions, state=None, dataset_stats=None, **kwargs) interface (see §2 The predict_action contract)
PayloadBuilder (client side) — payload_builders/<model>.py, registered via @register_payload_builder("<model_type>"), inherits from PayloadBuilder. Responsibilities:
- declare capabilities via type-annotated class attributes (state_encoding / action_encoding / action_dim / action_horizon / domain_id / unnorm_key, …; YAML overrides same-name fields)
- build(canonical, ctx): canonical dict → predict_action kwargs
- optionally reset / update_from_response / note_env_action for closed-loop feedback
ActionDecoder (optional, eval side) — action_decoders/, only when the model’s action_encoding has no decoder yet. Decoder selection:
- key auto-assembled as {action_encoding}_to_{action_space} (see §4 State / action semantics)
- source == target space → IdentityDecoder passthrough, no new code
- hits an already-registered key (e.g. ee6d_*) → selected automatically, no new code

Steps — integration flow:

Implement the model components — ModelFactory + PayloadBuilder (+ optional ActionDecoder), per the responsibilities above and the details in §3 Component details.
Implement the predict_action interface — the model must expose predict_action(images, instructions, state=None, dataset_stats=None); model-specific preprocessing and normalization live inside it, and the eval server calls it over RPC (see §2 The predict_action contract).
Write the eval config and scripts (examples/embodied/<model>/eval/) — model-specific YAML fields (state_encoding / action_encoding / domain_id, …) go under model:; model.model_type is required (no default). Verify each config point in §4 State / action semantics and §5 Engineering parameters.
Run a smoke (1 task × 1 episode, or a chain smoke) to validate the RPC / action semantics, then — with domain weights — task-success and full-scale eval.
Regression-test already-succeeding combinations (at least pi05 × LIBERO) after changing shared code (runner / adapter / decoder / bridge / generic policy).

Startup fail-fast: the Factory and the PayloadBuilder are registered independently, and a missing one fails fast on its own side — the eval server exits in build_model_spec when model.model_type has no registered factory, and the runner exits in build_payload_builder when it has no registered PayloadBuilder. Both must be registered for a run to start.

Runtime path — once a benchmark runner exists, do not write a per-model policy class: the server side is uniformly handled by GenericPredictActionPolicy for RPC / caching / statistics path / shape checking, and the model only implements predict_action in its factory:

loongforge_server.py
  -> <Model>ModelFactory.build(...)              # factories/<model>_factory.py
  -> GenericPredictActionPolicy(...)
  -> model.predict_action(images, instructions, state=None, dataset_stats=None, **kwargs)

2. The `predict_action` contract#

This section is the interface contract between the model author and the eval stack, aimed at the model owner who implements predict_action(). The single source of truth for the helper functions is loongforge/embodied/eval/servers/predict_action_interface.py: after refactoring, this file only retains the model-author contract — the PredictActionModel protocol, validate_predict_action_model, _filter_supported_kwargs, and call_predict_action; it does not contain any action-space decoding logic, which has been moved to loongforge/embodied/eval/action_decoders/ (see §4 State / action semantics).

2.1 Required signature#

def predict_action(images, instructions, state=None, dataset_stats=None, **kwargs):
    """Return decoder-ready actions as a float array."""
    ...

Parameter meanings:

Parameter	Typical type	Meaning
`images`	a batch-of-1 list-of-list (each element is a view array)	e.g. `[[view_primary, view_wrist]]`. The PayloadBuilder packs the per-camera dict into a view list (prefer `primary`/`head`; if `left` and `right` both exist → `[primary, left, right]`; otherwise append at most one more `wrist` → `[primary, wrist]`; single view → `[primary]`); the server then wraps it into batch-of-1.
`instructions`	`list[str]`	Batched language instructions (batch=1).
`state`	`None` or a numeric vector / array	The model-usable proprio the PayloadBuilder produces from `canonical["state_raw"]` according to `state_encoding` (may be `None`).
`dataset_stats`	`dict` or `None`	Loaded by the eval side from `server.dataset_statistics_path` and passed through; used internally by the model as needed for state normalization / action denormalization.
Extra kwargs	e.g. `domain_id`, `unnorm_key`	Emitted by the PayloadBuilder. Passed through only when the signature accepts them (explicitly named or `**kwargs`); unknown ones are dropped with a WARNING.

Images are a batch-of-1 list-of-list (each inner element is a view array), and instructions are list[str]. They are delivered to the model via the RPC v2 payload.

2.2 kwargs filtering#

call_predict_action inspects the signature before calling:

Signature contains **kwargs → all extra keywords are passed through.
Fixed signature → only the declared parameters are kept. Any field the PayloadBuilder emits but the model does not declare (such as a mistakenly emitted domain_id / unnorm_key) is dropped and a WARNING is logged, making a misconfigured PayloadBuilder visible in the logs rather than producing a “no error but wrong result” run.

If a model needs to consume a certain field, it must declare that parameter in the signature (or use **kwargs).

2.3 Where `state` comes from#

The adapter no longer encodes proprio; it only exposes the raw fields in canonical["state_raw"] (eef_pos / eef_quat / ee_ori_mat / joint / endpose / robot_obs, etc.); each model’s PayloadBuilder encodes them into the state kwarg according to its state_encoding ("" → no state; ee6d / aloha_pi / passthrough, etc.).

The PayloadBuilder’s build(canonical, ctx) returns the model kwargs; the runner adds RPC control fields around them:

model_kwargs = payload_builder.build(canonical_obs, ctx)  # {"images": [...], "instructions": [...], "state": ...}
rpc = {"episode_id": ..., "episode_step": ..., "disable_action_cache": ..., "return_action_chunk": ...}
rpc.update(model_kwargs)

canonical.state_raw  ->  PayloadBuilder(state_encoding)  ->  RPC payload.state  ->  predict_action(state=...)

2.4 Validation helpers#

loongforge/embodied/eval/servers/predict_action_interface.py

API	Purpose
`PredictActionModel`	Protocol
`validate_predict_action_model(model)`	Pre-call signature check
`_filter_supported_kwargs(func, kwargs)`	Drops kwargs the signature does not accept (one WARNING logged per dropped item)
`call_predict_action(model, images, instructions, state, dataset_stats, action_dim, **kwargs)`	validate → filter kwargs → call → reshape / truncate to `[H, action_dim]`

GenericPredictActionPolicy always calls through call_predict_action. Action-space decoding is not here; it is in action_decoders/.

validate_predict_action_model checks: predict_action exists and is callable; the required parameters images / instructions are present; the optional parameters state / dataset_stats can be accepted (as named parameters or via **kwargs).

# invalid: missing instructions, and cannot accept state/dataset_stats
def predict_action(self, images):
    ...

# valid
def predict_action(self, images, instructions, state=None, dataset_stats=None):
    ...

def predict_action(self, images, instructions, **kwargs):
    state = kwargs.get("state")
    dataset_stats = kwargs.get("dataset_stats")
    ...

2.5 Action output contract#

The model may return [D] / [H, D] / [B, H, D]; call_predict_action uniformly normalizes to [H, action_dim]:

Input shape	Behavior
`[D]`	→ `[1, D]`
`[H, D]`	Kept as a chunk
`[B, H, D]`	→ `[-1, D]` (single-request path)
Other ndim	`ValueError`
Last dim `< action_dim`	`ValueError`
Last dim `> action_dim`	Truncated to the first `action_dim` columns

action_dim comes from the model / YAML config (e.g. 7 for single-arm, 14 for RoboTwin joints, 20 for xvla ee6d). Truncation cannot replace correct action semantics.

Normalization / denormalization belongs to the model: the eval does not perform q01/q99, mean/std, or min/max outside the model. If the network outputs normalized actions, they must be denormalized inside predict_action using dataset_stats (and the training normalization mode). For example, pi05’s ACTION quantile normalization uses dataset_stats["action"].q01 / .q99; other LeRobot-family models may use mean/std or min/max. The return value should be in the model action space (i.e. the encoding declared by the PayloadBuilder’s action_encoding), and either match the environment after truncation (e.g. pi05 LIBERO 7D axis_angle), or be converted by an eval ActionDecoder / RoboTwin action_bridge (e.g. xvla 20D ee6d).

Action decoding on the eval side does not belong to predict_action: after the server returns the chunk, the runner applies the decoders in action_decoders/:

from loongforge.embodied.eval.action_decoders import build_action_decoder
from loongforge.embodied.eval.orchestrator.config import resolve_action_decoder_key

key = resolve_action_decoder_key(payload_builder, adapter)  # {action_encoding}_to_{action_space}
decoder = build_action_decoder(key)                         # empty key -> IdentityDecoder
env_actions = decoder(raw_chunk, ctx)                       # __call__(actions[H, D], ctx) -> env_actions

2.6 Local interface validation#

No GPU / weights needed; use this to validate the eval helpers before fully loading the model:

cd /workspace/LoongForge-VLA
PYTHONPATH=/workspace/LoongForge-VLA python - <<'PY'
import numpy as np
from loongforge.embodied.eval.servers.predict_action_interface import (
    call_predict_action,
    validate_predict_action_model,
)

class FixedSigModel:
    """pi05-style: extra kwargs are filtered out."""

    def predict_action(self, images, instructions, state=None, dataset_stats=None):
        return np.zeros((len(instructions), 4, 7), dtype=np.float32)

class KwargsModel:
    """xvla-style: domain_id is forwarded."""

    def predict_action(self, images, instructions, state=None, dataset_stats=None, **kwargs):
        assert kwargs.get("domain_id") == 6
        return np.zeros((4, 20), dtype=np.float32)

images = [[np.zeros((224, 224, 3), dtype=np.uint8)]]
common = dict(instructions=["pick up the cube"], state=None, dataset_stats=None)

m = FixedSigModel()
validate_predict_action_model(m)
print(call_predict_action(m, images=images, action_dim=7, do_sample=False, cfg_scale=1.5, **common).shape)

m = KwargsModel()
validate_predict_action_model(m)
print(call_predict_action(m, images=images, action_dim=20, domain_id=6, **common).shape)
PY

Expected output:

(4, 7)
(4, 20)

Comparison of the predict_action output of the real pi05 / xvla after loading via factory (random_init is enough for a contract check, but full correctness (unnorm, abs/delta, decode) still requires a YAML smoke / task-success run, see user_guide_en.md):

Model	Raw `predict_action`	After `call_predict_action`
pi05	`[B, action_horizon, max_action_dim]`, e.g. `(1, 50, 32)`	truncate last dim → `(50, 7)`
xvla	`[B, num_actions, real_action_dim]`, e.g. `(1, 30, 20)`	reshape + keep dim → `(30, 20)`

pi05 requires tokenizer_path (PaliGemma tokenize) even with random_init.
xvla requires a valid Florence processor/tokenizer directory (tokenizer_path); an empty path causes HF loading to fail.
The xvla factory converts the domain_id int → a LongTensor on the device; call_predict_action(..., domain_id=3) can just pass a YAML-style int.

2.7 Warmup and common errors#

Before the health endpoint is ready, the server may make one call first (images=[[np.zeros((224,224,3), uint8)]], instructions=["warmup"], state=None, dataset_stats=None). A failure only logs a warning, but this call must not corrupt weights or render the process unusable; prefer a lazy import that can complete safely.

Error	Cause
`TypeError: model must expose a callable predict_action(...)`	Missing method
`TypeError: ... missing required parameters: ['instructions']`	Wrong signature
`TypeError: ... cannot accept eval keyword parameters: ['state']`	No `state` and no `**kwargs`
`ValueError: ... unsupported action shape`	Not `[D]` / `[H,D]` / `[B,H,D]`
`ValueError: ... action dim X, expected at least Y`	Output dimension narrower than `action_dim`
The env has steps but the success rate is always 0	Usually the control mode (abs vs delta), a wrong ActionDecoder / bridge, or wrong unnorm — not a missing `predict_action`

Self-check for the model owner before delivery: predict_action(images, instructions, state=None, dataset_stats=None) is callable; validate_predict_action_model passes; returns [D]/[H,D]/[B,H,D] with last dim ≥ action_dim; denormalization (if any) happens inside predict_action, and the eval only passes through dataset_stats; warmup is safe; the factory is responsible for loading weights / tokenizer / processor and does not rewrite the benchmark dict observation.

3. Component details#

Factory / PayloadBuilder / ActionDecoder are summarized in §1 Overview; this section walks through each component in detail and covers the RoboTwin action_bridge.

3.1 ModelFactory#

Location: factories/<model>_factory.py; register with @register_factory("<model_type>"). Declare model_config_cls and implement build(model_cfg, server_args) -> PredictActionModelSpec:

model_config_cls: the typed config dataclass resolved from the YAML model: section (e.g. Pi05ModelConfig).
build(...): load the model — import, config, weights / random_init, device/dtype — and return PredictActionModelSpec(model=..., metadata={...}), where model implements predict_action (see §2 The predict_action contract).
The registry key must pair with a PayloadBuilder of the same model_type (both must be registered for the run to start — see the startup fail-fast in §1 Overview).

References: factories/pi05_factory.py, factories/xvla_factory.py, factories/groot_n1_6_factory.py.

3.2 PayloadBuilder#

Location: payload_builders/<model>.py; register with @register_payload_builder("<model_type>") and inherit from PayloadBuilder:

Capability class attributes: state_encoding / action_encoding / action_dim / action_horizon / domain_id / unnorm_key, … — YAML model: fields of the same name override them.
build(canonical, ctx) -> dict: canonical observation (from the adapter) → predict_action kwargs — image packing per the view policy, state per state_encoding, model-specific fields.
Optional closed-loop hooks: reset() / update_from_response() / note_env_action().

References: payload_builders/pi05.py, payload_builders/xvla.py, payload_builders/groot_n1_6.py.

3.3 ActionDecoder#

Components: action_decoders/ (base.py defines the ActionDecoder base class + IdentityDecoder + ACTION_DECODER_REGISTRY; ee6d.py holds the ee6d source-encoding decoders; joint.py holds the joint source-encoding decoders; rotation.py stores the rotation math). The orchestrator automatically assembles the decoder key from {payload_builder.action_encoding}_to_{adapter.action_space} (resolve_action_decoder_key), and when the source encoding == target space it returns an empty key → IdentityDecoder passthrough.

Registered keys (ACTION_DECODER_REGISTRY) auto-assembled as {action_encoding}_to_{action_space}:

key	Use
`ee6d_to_axis_angle`	xvla × LIBERO / ManiSkill: 20D EE6D → 7D (pos + axis-angle + grip)
`ee6d_to_simpler_abs_euler`	xvla × SimplerEnv WidowX: rot6d→euler + offset + grip mapping
`ee6d_to_calvin_abs`	xvla × CALVIN official absolute-pose protocol
`ee6d_to_euler` / `ee6d_to_quat`	Other EE variants
`pi05_aloha_robotwin`	pi05 × RoboTwin joint decoder (stateful; via bridge)
`ee6d_robotwin_ee_dual`	xvla × RoboTwin dual-arm ee decoder (via bridge)

pi05: action_encoding == adapter.action_space (e.g. axis_angle) → empty key → passthrough.
xvla: action_encoding: ee6d × each benchmark’s action_space → auto-select the key from the table above.

If the new model’s output is inconsistent with the environment’s native action space, register the corresponding decoder under action_decoders/ (@register_action_decoder("<encoding>_to_<space>")); do not write environment special-casing into the training-side predict_action.

3.4 `benchmark.action_bridge`#

Most benchmarks run through the standard runner chain: the eval runner drives the env, calls predict_action over RPC, and applies the decoded action. RoboTwin is different: the official protocol only exposes a policy-plugin interface — the official evaluator (script/eval_policy.py) owns the env (observation collection, stepping, success judgment) and reverse-calls the policy plugin. We therefore run the official evaluator as a subprocess, and action_bridge selects the (model_type, PayloadBuilder state_encoding, decoder key) wiring hosted inside the plugin (bridges/robotwin_policy.py), so the official evaluator can drive a LoongForge policy without touching the model’s default behavior.

Implementation: bridges/robotwin_policy.py (_BRIDGE_WIRING maps a bridge name to (model_type, payload-builder state_encoding, decoder key), assembling the shared adapter → PayloadBuilder → PolicyClient → ActionDecoder four-component chain).

bridge	Use
`pi05_aloha_14d`	pi05 RoboTwin official protocol (`Pi05PayloadBuilder(state_encoding="aloha_pi")` + `pi05_aloha_robotwin` decoder: adapt_to_pi + delta→abs, stateful)
`ee6d_dual`	xvla RoboTwin official protocol (`XVLAPayloadBuilder(state_encoding="ee6d_dual")` + `ee6d_robotwin_ee_dual` decoder; 20D EE, three views, `action_type='ee'`)

Protocol logic is placed in named bridge modes, avoiding changes to the model’s default behavior that would affect other benchmarks.

4. state / action semantics#

This chapter lists what to confirm when integrating a model. The per-model values are not duplicated here — they live with the model itself: the PayloadBuilder class attributes and comments in payload_builders/<model>.py (state_encoding / action_encoding / action_dim / action_horizon, and the supported encoding values), the factory, and that model’s eval YAMLs under examples/embodied/<model>/eval/configs/.

4.1 Action space and dimensions#

The action dimension changes with the benchmark protocol and cannot simply be hard-coded to LIBERO (a dual-arm benchmark such as RoboTwin needs a different protocol from single-arm LIBERO, which is what benchmark.action_bridge selects).

If the model’s action_encoding equals the env action_space, the decoder key is empty and an IdentityDecoder passes the action straight through; otherwise the decoder key is auto-assembled as {action_encoding}_to_{action_space}.

Points to confirm: the total number of model output dimensions, the semantic layout of each dimension (position / rotation / gripper), the rotation representation (axis-angle, 6D rotation, quaternion), and the target environment’s control interface (joint vs EE).

4.2 Control mode#

Confirm whether the model emits delta or absolute targets and whether the env is configured to match: benchmark.control_mode (delta / absolute / auto, where auto infers from the decoder key) plus, for some envs, a controller that must exist on the env side (e.g. SimplerEnv WidowX absolute EE, see patches/simplerenv/xvla.md).

It should be emphasized that an absolute pose cannot be crudely turned into a delta by “linearly subtracting” the current pose: axis-angle rotation does not satisfy linear subtraction, and the delta mode often has action scaling. A wrong control mode is the primary reason xvla initially had a 0 success rate on LIBERO.

4.3 Proprioceptive input#

Proprio (proprioception) is the robot’s own state — joint positions, end-effector pose, gripper state — as opposed to external perception (cameras). In this framework it is what the model receives as the state argument of predict_action.

The adapter no longer encodes proprio: it only exposes the raw EE / joint fields in canonical["state_raw"] (eef_pos / eef_quat / ee_ori_mat / joint / endpose / robot_obs, etc.); the encoding is done by each model’s PayloadBuilder according to model.state_encoding (the supported values are listed on each PayloadBuilder class).

Points to confirm:

Every slot must reproduce the training-time definition, value by value, from the official wrapper — not a physically similar quantity. A normalized gripper openness and a finger width in metres both “look like a gripper state” but land in completely different places after normalization, and nothing errors out.
Rotation layout (e.g. column-major vs interleaved 6D) — a wrong layout silently shifts the input distribution; this was the second reason xvla scored 0 on LIBERO.
Whether proprio should be the env’s measured state or a closed-loop feedback of the previous predicted action (the official client decides; the stateful encodings implement feedback via update_from_response / note_env_action).

Boundary: canonical["state_raw"] holds the raw fields (for the PayloadBuilder to encode + for trace/debug); the state kwarg produced by the PayloadBuilder’s build() enters predict_action(state=...) via RPC. Do not pass a nested dict directly as state to predict_action (unless the model explicitly declares that layout); prefer passing a flat float32 vector aligned with the training observation.state.

4.4 Normalization approach#

Normalization/denormalization stays inside the model (predict_action); the generic policy never unnorms (see the ownership convention in §2.5 Action output contract). What differs per model is only where the statistics come from — an external dataset_statistics.json via server.dataset_statistics_path, the model’s own action space, or config files shipped with the weights.

Points to confirm: the stats source and the normalization mode per key are read from the weights’ own config, not hand-derived from observed value ranges; and denormalization happens in exactly one place. A benchmark whose action ranges are small can pass while a wide-range benchmark fails — a passing benchmark does not prove the normalization is right.

4.5 Model-specific request fields#

Model-specific fields are declared by the PayloadBuilder and injected into the predict_action kwargs; the YAML writes them under the model: section. A multi-domain model, for instance, needs a domain/task id chosen per benchmark (the PayloadBuilder may default it from the benchmark name), and the factory converts it to whatever tensor type the model expects. A misconfiguration usually produces no error, but the action distribution is wrong — so treat these fields as part of the protocol and take their values from the official eval / production config, never invent them.

The same applies to a new model’s task embedding / domain embedding / special prompt, etc.: model: YAML → PayloadBuilder → RPC payload → factory → model, end to end. If the PayloadBuilder emits a kwarg the model signature does not accept, _filter_supported_kwargs drops it with a WARNING (see §2.2 kwargs filtering); use this to diagnose misconfigurations.

4.6 Number and order of image views#

View packing follows these rules (not a fixed 2 views):

There must be a primary or head;
If left and right both exist → 3 views [primary, left, right] (RoboTwin / X-VLA official);
Otherwise append at most one more wrist (or a standalone left/right) as the 2nd view.

(Implemented in payload_builders/pi05.py::_pack_images, which the xvla PayloadBuilder reuses.)

The adapter declares its camera set with the cameras class attribute (e.g. LIBERO ("primary", "wrist")), and the PayloadBuilder packs the model’s expected view list from canonical["images"] (a per-camera dict) per the rules above. The view count therefore follows the adapter, not a per-model setting: the model side must support a dynamic num_images = len(images[0]) and must not hardcode num_images=2 or 3 (otherwise it breaks other benchmarks).

Points to confirm: the number, order, and resolution of cameras at training time, and whether they are flipped (LIBERO agentview vertical flip is handled uniformly by the adapter).

Views are dynamically packed by the PayloadBuilder from the obs; do not rely on fabricated YAML fields to control the number of views.

5. Engineering parameters#

Load timeout (start_timeout_sec): xvla cold start can be >900s, and task-success configs commonly use 2400; 900 is usually enough for pi05; a random_init chain smoke can be shorter.
processor / tokenizer: pi05 needs an external paligemma (tokenizer_path); xvla’s processor is in the weight directory (processor_path / tokenizer_path are often the same directory).
server.random_init: chain smoke when there are no weights; paired with an empty ckpt_path.
Ports: each run uses an independent port / health_port to avoid chunk-cache cross-contamination.
GPU: usually one card per policy server; running eval tasks serially is more reliable.
Environment separation: the orchestrator uses the benchmark conda; server.python uses the model server environment.
SAPIEN (SimplerEnv / RoboTwin / ManiSkill): besides nvidia-smi, use vulkaninfo to confirm deviceName=NVIDIA (not llvmpipe); set LD_LIBRARY_PATH and VK_ICD_FILENAMES.

Eval parameters (max_steps / num_steps_wait, etc.) must be configured according to the original eval protocol + the current smoke intent, not mixed:

pi05 × LIBERO: smoke commonly uses max_steps: 300; full-scale long-horizon suites such as libero_10 are recommended to be higher (e.g. 520). A max_steps that is too small will judge in-progress episodes as failures.
xvla × LIBERO: original horizon 800, num_steps_wait: 10; smoke can be 1 task × 1 ep, but keeping 800 is still recommended so long tasks are not truncated.
xvla × SimplerEnv WidowX: official max_steps: 1200 (task-success config).
Chain smoke (random_init): deliberately short step counts (e.g. 20–30), only to prove RPC, not counted as a result.

runner semantics: when benchmark.max_steps > 0, it takes precedence over the suite default (not bounded by its cap).

6. Development notes#

Keep model framework logic in eval/factories/<model>_factory.py. Do not add model-specific code to loongforge_server.py.
Do not modify training-tree LoongForge source for eval-specific compatibility.
Prefer adding YAML configs over command-line parameter sprawl.
Use model.backend in YAML for backend selection and benchmark.name for benchmark selection.
New model factories should ensure their predict_action() tolerates a warmup call with a zero-filled dummy image and an empty instruction string (_warmup_model in loongforge_server.py runs this before serving).
After changing shared runner / adapter / payload builder / action decoder / bridge / generic policy code, regression-test at least pi05 × LIBERO.

Guide to Integrating a New Model into the Eval System

Contents