skill-test-execute

To run fixture-based skill testing, provide the target skill directory and the scenario directory name. The runner executes the skill in a copy-only sandbox seeded from before/ directories and compares the resulting filesystem against after/ oracles to emit a PASS/FAIL/BLOCKED verdict.

Sandbox isolation in skill testing executes the target skill in a copy-only environment seeded exclusively from before/ fixtures. This strict isolation prevents oracle leakage and fixture contamination by ensuring the tested skill cannot access the after/ oracles during execution.

Filesystem diff evaluation compares the sandboxed execution result against the scenario oracle. It applies contract-aware semantic judgment by referencing the target skill's SKILL.md to classify diffs as blocking failures, non-behavioral drift, or runner ambiguity, ultimately emitting a PASS, FAIL, or BLOCKED report.

Yes, TEST.md contract validation is required for sandbox execution. The testing process requires a .tests/TEST.md file to define before/after semantics, oracle isolation, and comparison rules, alongside a SKILL.md presence check to ensure the target skill's contract is properly validated.

No, the fixture runner does not generate persistent reports or mutate fixtures. It produces a single compact evaluator report in chat based on the sandboxed execution, ensuring your before/ seeds and after/ oracles remain completely uncontaminated for deterministic scenario testing.

You should not use this copy-only sandbox approach for non-deterministic scenario testing or if your evaluation requires persistent report generation. It is designed strictly for deterministic scenarios where only chat-based evaluator output is allowed and filesystem diffing determines the verdict.