evaluating-cosmos-policy

Run the official cosmos_policy.experiments.robot.libero.run_libero_eval module with the LIBERO checkpoint, dataset statistics, and T5 embeddings paths. Start with a 1-trial smoke eval on libero_10, then scale to 50 trials across all four suites.

Install the Cosmos-compatible RoboCasa fork, run setup_macros and download_kitchen_assets, then invoke cosmos_policy.experiments.robot.robocasa.run_robocasa_eval with a task name like TurnOffMicrowave and a fixed object instance split.

A single A40 or A100 GPU with roughly 16 GB VRAM handles LIBERO evaluations, while RoboCasa needs about 18 GB. Smoke evals take 5-15 minutes; full benchmarks run 2-8 hours depending on suite and trial count.

Rendering fails when CUDA_VISIBLE_DEVICES and MUJOCO_EGL_DEVICE_ID point to different GPUs or EGL variables are unset. Export MUJOCO_GL=egl, PYOPENGL_PLATFORM=egl, and align both device IDs to the same GPU index.

Use fine-tuning-openvla-oft for OpenVLA-based policies and fine-tuning-serving-openpi for Physical Intelligence pi0 models. This skill covers only simulation evaluation of Cosmos Policy, not training, fine-tuning, or real-robot deployment.