skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

i4h-workflow-dataset-annotate

Annotate robotic manipulation episodes in HDF5 recordings with VLM-based success labels.

3.2k|370|Updated Feb 25, 2026
Search Tags:#robotics#annotation#vllm#vlm#HDF5#isaac-for-healthcare#dataset filtering
One-click install
npx skills add https://github.com/NVIDIA/skills --skill i4h-workflow-dataset-annotate
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: i4h-workflow-dataset-annotate
Source: https://github.com/NVIDIA/skills/tree/main/skills/i4h-workflow-dataset-annotate
Command: npx skills add https://github.com/NVIDIA/skills --skill i4h-workflow-dataset-annotate

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Robotics teams recording teleoperation or policy-evaluation episodes need to know which episodes actually satisfy the task description before filtering demonstrations or gating finetuning datasets, and manually reviewing every episode is slow and error-prone.

Core Features & Use Cases

  • VLM-Based Episode Annotation: Sends episode frames to an OpenAI-compatible vision model endpoint (default Qwen/Qwen3-VL-8B-Instruct) to judge each episode against the environment's task description from the env YAML config.
  • Offline and Live Modes: Annotate recorded HDF5 files offline with optional filtered-dataset output, or judge live camera frames from a running policy/Arena session over Zenoh with success-gate thresholds.
  • Use Case: After a scissor pick-and-place data collection run, ask your agent to annotate all recorded episodes; it resolves the latest HDF5 recording, runs the annotator, and reports success/failure counts plus a filtered HDF5 containing only passing demonstrations.

Quick Start

Ask your agent to run VLM annotation on all recorded episodes of the latest scissor pick-and-place recording and summarize the success and failure counts.

Frequently Asked Questions about i4h-workflow-dataset-annotate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I annotate robot episodes with a VLM?▼

Run the annotator's offline mode against an HDF5 recording, pointing --base-url and --model at an OpenAI-compatible vision model endpoint. The skill resolves the latest recording automatically, writes annotations.jsonl, and can produce a filtered HDF5 of successful episodes.

How do I filter robot demonstrations by task success?▼

Pass --filter <path> to the offline annotation command to write a filtered HDF5 containing only episodes the VLM judged successful. Success is determined against the task description in the environment YAML, overridable with --task-description.

Can I use a text-only LLM for episode annotation?▼

No. Annotation sends image inputs, so the endpoint must serve a vision-language model such as Qwen/Qwen3-VL-8B-Instruct. Text-only or code models like qwen3-coder-next return bad-request errors and cannot be used.

Why does annotation fail with connection refused at localhost:8000?▼

Connection refused means no vLLM server is running at VLM_BASE_URL. Start one with annotator/vllm.sh ensure, or set VLM_BASE_URL and VLM_MODEL to point at an already-running vision model server.

When should I use live annotation mode instead of offline?▼

Use live mode only when a policy or Arena session is already running and you want real-time judging of camera frames over Zenoh. For recorded data, offline HDF5 annotation is the correct path; annotation is skipped during validation unless explicitly requested.

Related Skills

Explore More Software Engineering Skills→
isaac-simisaac-sim
4.0k

urdf-mjcf-to-usd-conversion

Convert URDF and MJCF robot descriptions to USD assets for Isaac Sim and Isaac Lab.

Official
Advanced
tadod12tadod12

fine-tuning-serving-openpi

Fine-tune and serve OpenPI robot policies across ALOHA, DROID, and LIBERO environments.

Community
Advanced
zerwizzerwiz

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM and OpenAI-compatible endpoints.

Community
Advanced
nicsuzornicsuzor
3

ground-truth

Create and refine ground truth labels for evaluation datasets with documented reasoning.

Community
Intermediate