Evaluation Framework & Protocol

Evaluation Framework & Protocol#

This page describes the LoongForge offline evaluation framework: its architecture, the runtime components, and the wire protocol between the benchmark client and the policy server. It is written for code contributors — model integrators and maintainers who touch eval code. End users only need the user guide and the benchmark pages.

Overview#

The eval module runs a policy against a simulation benchmark from a single YAML config. It launches the benchmark simulator and an independent model server as separate processes connected by a WebSocket + msgpack-numpy RPC protocol. The model and the benchmark are decoupled: benchmark code never imports model code, and vice versa. (The architecture diagram lives in the module overview Architecture section.)

Components#

Component

Responsibility

adapters/<benchmark>.py

env observation → canonical dict (state_raw); declares matching capabilities (action_space, default_fps, cameras)

payload_builders/<model>.py

canonical dict → predict_action(**kwargs); declares model capabilities (state_encoding / action_encoding / action_dim / …)

model.predict_action (server)

model inference; owns all model-specific normalization/unnormalization

action_decoders/<encoding>.py

raw model action chunk → benchmark env action space; identity match yields a no-op IdentityDecoder

factories/<model>_factory.py

thin model load/build, registered via @register_factory("<model_type>")

transport/

PolicyClient (WebSocket client), msgpack-numpy serialization, RPC server

orchestrator/

unified YAML entry (run.py), server manager, runners, config assembly

At runtime every benchmark drives the same four-stage chain:

Adapter.obs_to_canonical(env_obs)      # env obs -> canonical dict
  -> PayloadBuilder.build(canonical)   # canonical -> predict_action kwargs
  -> model.predict_action(**kwargs)    # RPC to the policy server
  -> ActionDecoder(action_chunk, ctx)  # raw model chunk -> env action space

The ActionDecoder key is auto-composed at startup as "{model.action_encoding}_to_{adapter.action_space}"; an identity match yields an empty key and an IdentityDecoder no-op.

Message flow#

The client and the server exchange msgpack-numpy-encoded PolicyRequest / PolicyResponse envelopes (types in protocol/schema.py, PROTOCOL_VERSION = "1.0"):

# request
{"type": "ping" | "reset" | "infer" | "predict_action",
 "request_id": str, "protocol_version": "1.0", "payload": {...}}
# response
{"status": "ok" | "error", "ok": bool, "type": str,
 "request_id": str, "data": {...}, "error": {...}}

  1. Connect — the client opens a WebSocket to the server; the server first sends its ServerMetadata (protocol_version, action_chunk_size, available_unnorm_keys, …). A protocol_version mismatch aborts the connection.

  2. Per step — the client sends predict_action with the PayloadBuilder-produced kwargs as payload; the server replies with the raw action chunk (plus model metadata) in data.

  3. ping — liveness check; reset — clears the server’s per-episode action chunk cache.

  4. Close — the client closes the WebSocket; the orchestrator terminates the server process.

The protocol is framework-agnostic: any policy only needs a compatible policy server.

PolicyClient interface#

transport/client.py provides PolicyClient:

  • ping() — liveness check

  • request(type, payload, request_id) — generic request; message types ping / reset / infer / predict_action

  • metadata — server metadata (protocol version, action dim, unnormalization mode, plus factory-declared fields such as action horizon / tokenizer path)

Serialization notes:

  • WebSocket payloads do not serialize PIL objects directly; convert images to np.ndarray on the client side and restore to PIL inside the framework if the model requires it.

  • The PolicyClient strips HTTP(S)/ALL proxy environment variables before connecting.

Running the policy server standalone#

The server entry is loongforge_server.py, launched by the orchestrator from the YAML config (--config <yaml>; the server: section holds host/port, python, ckpt_path, use_bf16, …). For debugging, copy an existing YAML and set model.backend: mock to run a pipeline-only check without model weights.