eval-harness

You can automate pytest feature testing with structured JSON output by running a pipeline simulation that validates features against known inputs and expected results, generating structured JSON to compare actual versus expected behavior.

End-to-end feature evaluation before a pull request is best handled by running pytest tests alongside a pipeline simulation, using fixture data and expected outputs to verify correctness and produce structured JSON results for quick review.

Yes, pytest feature evaluations require a defined evals/ directory structure to locate features, fixtures, and expected results, along with an eval runner configuration in .claude/project.json to automate the workflow.

You can use fixture data to validate modified features by running pytest tests that compare known inputs against expected outputs, applying layered validation to ensure both logic correctness and end-to-end pipeline behavior.

Pipeline simulation fails to generate structured JSON results when the eval runner configuration in .claude/project.json is missing or the evals/ directory structure cannot properly locate the required features, fixtures, and expected outputs.

Eval-harness works for CI validation and quick feature verification by automating end-to-end feature evaluation through pytest tests and pipeline simulation, producing structured JSON results that compare expected versus actual outputs.