Design evaluation frameworks for LLMs and agents with rubric-based grading.
npx skills add https://github.com/Sokoliem/ultraprompt --skill llm-eval-design
Please help me install this Agent Skill. Skill: llm-eval-design Source: https://github.com/Sokoliem/ultraprompt/tree/main/skills/llm-eval-design Command: npx skills add https://github.com/Sokoliem/ultraprompt --skill llm-eval-design
Lays out disciplined processes to design, implement, and validate evaluation schemes for LLMs and agents, turning vague assessment into structured, repeatable tests.
Design an eval suite for a given LLM capability using golden, adversarial, and rubric cases and validate the grader with human labels.
High-intent search queries and answers about installing and using this skill.
To design an LLM evaluation framework, structure your tests into golden cases, adversarial cases, and rubric-based grading to probe capability, robustness, and scoring stability across offline and online scenarios.
Rubric-based grading for LLMs is an evaluation method that uses structured scoring criteria to assess model outputs. It validates grader stability against human labels to detect hallucination, drift, and capability regressions.
Validate LLM graders by establishing threshold baselines and comparing grader outputs against human-labeled golden cases. This ensures meaningful progress signals and detects regressions over time.
Yes, you can integrate LLM evaluation into CI pipelines using reproducible wiring and documentation. This enables automated regression testing, evidence collection, and dashboards for ongoing model experiments.
Track baseline metrics across grounding, hallucination, drift, latency, and cost. Use golden and adversarial cases to probe these capabilities and establish threshold baselines for detecting regressions.
Structure benchmark datasets using a three-tier approach: golden cases for baseline capability, adversarial cases for robustness, and rubric cases for scoring stability to ensure comprehensive agent evaluation.
DhumitechAutomate AI agent evaluation and benchmarking with repeatable test pipelines.
Sheldon-92Automate AI evaluation workflows for benchmarking, regression testing, and safety assessment.
DhumitechInstrument LLM pipelines for latency, token usage, and hallucination tracking with Langfuse.
yanochka11Build browser-based annotation interfaces for reviewing LLM traces with pass/fail labels.
