Validate agent skills with linting and LLM-based calibration drift detection.
npx skills add https://github.com/alex-voloshin-dev/ai-skills --skill eval-alex-voloshin-dev
Please help me install this Agent Skill. Skill: eval Source: https://github.com/alex-voloshin-dev/ai-skills/tree/main/plugin/skills/eval Command: npx skills add https://github.com/alex-voloshin-dev/ai-skills --skill eval-alex-voloshin-dev
This skill addresses the challenge of ensuring agent reliability by providing a structured harness to detect calibration drift, rubric regressions, and linting errors before code reaches production.
Use the eval skill to run the Tier 1 linters and Tier 2 calibration smoke tests across all available skills.
High-intent search queries and answers about installing and using this skill.
Testing agent skills for calibration drift requires a multi-tier evaluation harness that runs static linting and LLM-based calibration checks against established rubrics to detect regressions before production deployment.
LLM-based calibration in quality assurance uses a judge model to score agent outputs against defined rubric samples, detecting drift in prompt-following consistency and output quality during pre-release validation workflows.
Yes, executing LLM judge-based scoring for agent skill evaluation requires Anthropic API access and the official Python SDK to perform calibration tests and detect rubric adherence issues.
Yes, Tier 1 linting performs automated schema, frontmatter, and syntax checks on agent skills without requiring LLM calls, providing a fast first-pass validation layer before deeper calibration testing.
The best way to validate multiple agent skills pre-release is running a full evaluation pass that combines Tier 1 automated linting with Tier 2 LLM-based calibration smoke tests across all available skills to catch regressions.
Agent skill evaluation fails on schema and syntax checks when automated linting detects frontmatter misconfigurations or structural errors, preventing downstream LLM-based calibration tests from executing reliably.
bykoleksii-hardoApply advanced scoring, bias mitigation, and confidence calibration to LLM outputs.
Sheldon-92Automate AI evaluation workflows for benchmarking, regression testing, and safety assessment.
enigmaicon-engScore or compare LLM outputs with rubric-based LLM-as-judge evaluation and structured JSON results.
raphaelmansuyImplement agent evaluation and safety gates using MLflow 3.x scoring and tracing workflows.
