Evaluate AI workflows with code-based graders and structured reporting.
npx skills add https://github.com/Steffen025/pai-opencode --skill evals
Please help me install this Agent Skill. Skill: Evals Source: https://github.com/Steffen025/pai-opencode/tree/main/.opencode/skills/Evals Command: npx skills add https://github.com/Steffen025/pai-opencode --skill evals
The Evals Skill provides a structured framework to evaluate AI agent workflows—capturing transcripts, tool usage, and multi-turn interactions—so you can compare models, prompts, and configurations reliably.
Run an evaluation by creating a use case, adding test cases, and running the evaluator through the CLI: bun run ~/.opencode/skills/Evals/EvalServer/cli-run.ts --use-case <name> --model <model>
High-intent search queries and answers about installing and using this skill.
AI agent workflows are evaluated using code-based graders for deterministic checks on format, code, state, and tools, ensuring reproducible scoring across multi-turn interactions and tool usage transcripts for reliable model comparisons.
Model-based rubrics support nuanced AI evaluation judgments through natural-language assertions. This allows flexible grading of complex interactions where deterministic scorers fall short, capturing subjective quality metrics in structured results.
Automated benchmarks run through trial-based evaluation capturing pass@k and pass^k metrics. Create a use case, add test cases, and execute the evaluator via the CLI with specified model parameters to generate comparison reports.
Comparing models and prompts reliably requires a structured framework capturing transcripts and tool usage. Trial-based evaluation with deterministic and model-based graders produces structured results for easy analysis and reporting across configurations.
The AI evaluation framework operates without external dependencies and uses a CLI runner. Execute evaluations by invoking the run script with use-case and model parameters to generate structured evaluation outputs.
VjayRamDefine DSPy signatures and modules with deterministic evaluation across splits.
tadod12Evaluate code-generation models with the BigCode Evaluation Harness.
pradyumnasagarAutomate inter-skill data transfer and provenance capture via JSON Material Passports.
agentic-inBenchmark LLM performance across 60+ academic benchmarks with reproducible scoring.
