What problem does it solve? Setting up a new realtime eval in the openai-cookbook repository requires choosing the correct harness, creating prompt, tools, and data files, writing a README, and validating everything with smoke and full eval runs, which is tedious and error-prone to do by hand. ## Core Features & Use Cases - Harness Selection Guidance: Helps choose between the crawl (single-turn text-to-TTS), walk (audio replay or generated audio), and run (multi-turn simulation with tool mocks) harnesses based on the evaluation goal. - Automated Scaffolding: Runs a scaffold script to generate the eval folder with system prompt, tools.json, starter datasets, and a README containing run commands and data contracts. - Realistic Starter Data: Authors use-case-grounded starter rows or simulations instead of generic placeholders, then validates with smoke evals, full evals, and pytest. - Use Case: A developer wants to evaluate a customer-support voice agent's tool-calling behavior; the skill asks for the scenario, recommends the crawl harness, scaffolds the folder, writes three realistic starter rows, and verifies everything runs. ## Quick Start Ask the assistant to bootstrap a new realtime eval for your scenario, providing the eval name, goal, system prompt, tools, and data source so it can pick a harness and scaffold the folder.
