Design evaluation plans for AI/LLM features with scorers, datasets, and baselines.
npx skills add https://github.com/samcdavid/dotfiles --skill my-eval-plan
Please help me install this Agent Skill. Skill: my-eval-plan Source: https://github.com/samcdavid/dotfiles/tree/main/claude/skills/my-eval-plan Command: npx skills add https://github.com/samcdavid/dotfiles --skill my-eval-plan
Design evaluation plans for AI/LLM features before building them, ensuring structured criteria, guardrails, and measurable success across multiple platforms.
Provide a feature description or ticket, and I will generate a complete eval plan including dimensions, scorers, datasets, baselines, and an execution plan.
High-intent search queries and answers about installing and using this skill.
To design an LLM evaluation plan, define evaluation dimensions like quality, safety, latency, and robustness, then establish measurable success criteria and baseline targets before building the actual evals.
The best way to structure pre-deployment datasets is to formulate a dataset strategy that includes golden sets, edge cases, production samples, and adversarial tests to ensure comprehensive coverage and robustness.
Setting up LLM-judge scorers involves designing automated, LLM-judge, and human review scorers, alongside calibration workflows to consistently measure evaluation dimensions across your experiments.
Yes, this platform-agnostic approach works with Braintrust, LangSmith, custom harnesses, or manual reviews, providing an end-to-end execution plan for both offline and online pre-deployment evaluations.
Yes, you can generate a complete execution plan for offline and online AI evals by providing a feature description or ticket, which produces scorer definitions, dataset strategies, and baseline targets.
Sir-chawakornBuilds an LLM evaluation harness using golden datasets and code-based scorers.
maruakshayAssess adversarial robustness of AI systems against evasion attacks.
jxtngxConvert LangSmith trace logs into structured JSON evaluation datasets.
Zhow01Assess and benchmark LLM outputs across automated metrics, human judgments, and safety checks.
