eval

Evaluate IsaacSim checkpoints by submitting run_eval MCP tool jobs.

Updated Aug 27, 2026

Search Tags:#evaluation #checkpoint #cluster #isaacsim #ml-experiments #run_eval

Ready to run

npx skills add https://github.com/kang-jaehyun/Mac-the-researcher --skill eval-kang-jaehyun

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval
Source: https://github.com/kang-jaehyun/Mac-the-researcher/tree/main/.claude/skills/eval
Command: npx skills add https://github.com/kang-jaehyun/Mac-the-researcher --skill eval-kang-jaehyun

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Enable users to evaluate a trained model's performance by running IsaacSim simulations on a given checkpoint, helping validate improvements after training.

Core Features & Use Cases

Identify the checkpoint to evaluate (job ID, path, or HF repo ID).
Determine eval parameters (task name, data config, number of episodes, eval set).
Choose the appropriate cluster/partition (prefer skt / l40s-gpu; sync if needed).
Ensure rlwrld_isaac is available on the target cluster.
Submit evaluation using the run_eval MCP tool and report results.

Quick Start

Identify your checkpoint, ensure rlwrld_isaac is available on the target cluster, and submit the evaluation using the run_eval MCP tool.

Frequently Asked Questions about eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate a trained checkpoint with IsaacSim?▼

Evaluate IsaacSim checkpoints by identifying the checkpoint via job ID, path, or HF repo ID, then submitting the evaluation through the run_eval MCP tool on a target cluster. This measures model performance by running IsaacSim simulations to validate improvements after training.

What clusters are supported for running IsaacSim evaluations?▼

IsaacSim evaluations support local, skt L40S, and rlwrld cluster setups, preferring the skt / l40s-gpu partition for running simulations. The tool syncs if needed to ensure the target environment is ready.

Do I need rlwrld_isaac installed to evaluate IsaacSim checkpoints?▼

Yes, rlwrld_isaac must be available on the target cluster to submit an IsaacSim checkpoint evaluation. This environment is required along with checkpoint access and the run_eval MCP tool to execute the simulation.

What evaluation parameters are needed for an IsaacSim simulation?▼

IsaacSim simulation evaluations require specifying the task name, data configuration, number of episodes, and the evaluation set. These parameters define how the trained checkpoint is tested during the simulation run.

Can I evaluate checkpoints stored in a Hugging Face repository?▼

Yes, you can identify the checkpoint to evaluate using a job ID, local path, or Hugging Face repository ID. The evaluation tool accesses the specified checkpoint to measure model performance across configured clusters.

Why is my IsaacSim checkpoint evaluation not working on the cluster?▼

IsaacSim checkpoint evaluations fail if rlwrld_isaac is not installed on the target cluster, the checkpoint is inaccessible, or the run_eval MCP tool is unavailable. Ensure all dependencies are configured on the skt L40S or rlwrld cluster before submitting.