holdout-validation

Author holdout scenarios under .copilot/holdout/ for Guardian evaluation.

Updated Apr 12, 2026

Search Tags:#specification #review #guardian #scenarios #workflows #architect #holdout

Ready to run

npx skills add https://github.com/karim-bhalwani/agentic-harness --skill holdout-validation

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: holdout-validation
Source: https://github.com/karim-bhalwani/agentic-harness/tree/main/skills/holdout-validation
Command: npx skills add https://github.com/karim-bhalwani/agentic-harness --skill holdout-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Structural separation of test authorship from code authorship. Produces behavioral acceptance scenarios that implementation agents cannot see, evaluated independently by Guardian. Use when designing acceptance criteria, validating implementations against intent, or enforcing holdout-set discipline. DO NOT USE FOR: unit test writing (use guardian), implementation (use implementer), verifying task completion (use verification-before-completion), or general test strategy.

Core Features & Use Cases

Holdout scenario authoring: Architects define intent-level holdouts separate from code.
Guardian review compatibility: Guardians evaluate implementations against holdout scenarios without access to the code.
Workflow governance: Enforces structural separation to prevent model leakage and test gaming.

Quick Start

Architect writes 3–10 holdout scenarios per feature and stores them under .copilot/holdout/ with filenames like HOLDOUT.md.

Frequently Asked Questions about holdout-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I separate test authorship from code authorship to prevent model leakage?▼

Holdout validation separates test authorship from code authorship by storing behavioral acceptance scenarios in a restricted .copilot/holdout/ directory. This structural separation prevents implementation agents from seeing the tests, thereby stopping model leakage and test gaming.

What is holdout-based spec integrity in software specification workflows?▼

Holdout-based spec integrity is a discipline where architects author intent-level behavioral scenarios independently from the code. Guardian then evaluates implementations against these hidden holdout scenarios to verify true intent without exposing the tests to the implementation agent.

How do I create and store holdout scenarios for acceptance criteria validation?▼

Architects author 3 to 10 intent-based holdout scenarios per feature and save them under the .copilot/holdout/ path using a HOLDOUT.md naming convention. This isolates the acceptance criteria from the main codebase for independent Guardian review.

When should I not use holdout validation scenarios?▼

Do not use holdout validation for writing unit tests, performing code implementation, verifying general task completion, or designing general test strategy. It is strictly for enforcing holdout-set discipline and validating implementations against intent during specification and review workflows.

Can I use holdout validation to evaluate implementations without giving Guardians access to the code?▼

Yes, Guardian review compatibility allows Guardians to evaluate implementations strictly against holdout scenarios without needing access to the underlying code. This ensures the review process focuses entirely on behavioral intent rather than implementation details.

Does holdout validation require specific workflow roles to function properly?▼

The workflow governance requires an Architect to author the holdout scenarios and a Guardian to evaluate them. This enforces structural separation between specification and review, ensuring the implementation agent cannot access or game the hidden acceptance tests.