Evaluate AI agent performance using cost per successful task and trajectory reliability metrics.
npx skills add https://github.com/thorwhalen/ek --skill ek-dev-agents
Please help me install this Agent Skill. Skill: ek-dev-agents Source: https://github.com/thorwhalen/ek/tree/main/skills/ek-dev-agents Command: npx skills add https://github.com/thorwhalen/ek --skill ek-dev-agents
This skill addresses the complexity of evaluating AI agents by shifting the focus from token-based costs to cost-per-successful-task, ensuring reliable and economically sound agent performance.
Use the ek-dev-agents skill to initialize a new agent evaluation suite with cost-sensitive metrics and a regression gate for your current task set.
High-intent search queries and answers about installing and using this skill.
Agent performance is measured by tracking trajectory reliability and calculating the cost per successful task, shifting focus from raw token usage to economically sound outcomes.
Reliability estimation uses pass-at-k and pass-hat-k pure functions to distinguish between an agent's base capability and its production-grade reliability across stochastic trials.
Regression gating prevents performance regressions by applying variance-aware statistical gates to stochastic trial groups, ensuring consistent agent evaluation results across tests.
LLM-as-judge validation pipelines integrate with cost-aware evaluation harnesses to assess tool-call grammars and measure the specific cost of errors in agent arguments.
Variance-aware regression testing groups stochastic trials to analyze cost-sensitive errors, preventing false positives when evaluating agent trajectory reliability and performance.
You can initialize a new agent evaluation suite with cost-sensitive metrics and a regression gate, directly evaluating tool-call grammars and argument error costs.
VjayRamDefine DSPy signatures and modules with deterministic evaluation across splits.
nguyenpv1980-wqDesign performance measurement harnesses with baselines and variance-aware regression gates.
BagelHoleInstrument AI agents with trace IDs and child spans for LLM calls.
ryasrkRoute LLM API requests by task complexity with budget tracking and retry logic.
