Automate Claude Code session evaluations with code, model, and human graders.
npx skills add https://github.com/cescrafli/compyrasion --skill eval-harness-cescrafli
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/cescrafli/compyrasion/tree/main/skills/eval-harness Command: npx skills add https://github.com/cescrafli/compyrasion --skill eval-harness-cescrafli
Formal evaluation is essential for AI-assisted development to prevent regressions and misbehavior. It provides a structured way to define, run, and track evaluations for Claude Code sessions, ensuring consistent quality and measurable progress.
Define a new eval as feature and run the evaluation suite to produce a report.
High-intent search queries and answers about installing and using this skill.
You can automate regression evals for AI-assisted coding sessions by defining an evaluation suite that applies deterministic code-based graders and model-based graders. This tracks behavior stability after changes to prevent regressions.
Eval-driven development is a formal evaluation process needed when integrating AI-assisted development to prevent misbehavior and regressions. It provides a structured way to define, run, and track evaluations for measurable progress.
Yes, model-based graders accommodate open-ended judgments for AI evaluations. The framework supports code-based grading for deterministic checks, model-based grading for flexible evaluations, and human grading for manual reviews.
You can track baseline performance and pass@k metrics by running configurable evaluation workflows that quantify reliability over time. This enables measurable progress tracking across different model versions and capability evaluations.
To set up capability evals, define a new eval as a feature and run the evaluation suite to produce a report. This verifies new features or model capabilities before integration into your AI-driven development workflow.
istinataFTSAnalyze Claude Code session JSONL logs for token usage and savings.
oabdelmaksoudImplement evaluation-driven development with code and model-based evaluators.
TruCol270Evaluate Claude Code sessions with configurable graders and pass@k metrics.
ThejanaJayalathDefine and run capability and regression evals for AI-assisted development sessions.
