Benchmark and validate AI models across accuracy, safety, cost, and latency.
npx skills add https://github.com/bipinks/ghost-office --skill ai-evaluation-bipinks
Please help me install this Agent Skill. Skill: ai-evaluation Source: https://github.com/bipinks/ghost-office/tree/main/.claude/skills/ai-evaluation Command: npx skills add https://github.com/bipinks/ghost-office --skill ai-evaluation-bipinks
Evaluating AI/LLM systems across accuracy, safety, cost, and latency to ensure trustworthy performance and actionable insights.
Run the evaluation suite on your LLM deployment to measure accuracy, safety, latency, and cost.
High-intent search queries and answers about installing and using this skill.
Automated evaluation pipelines run eval cases against LLM deployments, apply rubric-based scoring for accuracy and safety, and compare model outputs. You construct eval datasets, trigger red-teaming, and capture latency or cost metrics across runs.
LLM-as-judge scoring uses a language model to automatically grade another model's outputs against defined rubrics. It evaluates dimensions like accuracy, relevance, and safety without requiring manual human review for every eval case.
Bias detection and hallucination checks are integrated into the evaluation suite. You run automated red-teaming and safety evaluations to identify biased outputs and factual hallucinations, ensuring trustworthy AI performance before deployment.
Yes, you can run A/B tests to compare different LLMs across accuracy, safety, cost, and latency. The evaluation pipelines apply standardized rubrics to eval cases, generating actionable insights to benchmark and validate models.
Production monitoring hooks integrate with model monitoring and reporting to track AI safety and performance continuously. You apply automated evaluation dimensions and bias checks to live deployments, ensuring ongoing trusted performance.
You need to run an AI safety benchmark and red-teaming suite when evaluating LLMs before deployment or during ongoing production monitoring. It validates accuracy, detects biases, and prevents hallucinations to ensure trustworthy performance.
vecearAutomate LLM evaluation with golden datasets, rubrics, thresholds, and regression reports.
chiruu12Monitor production apps for errors and regressions after deployment.
arsityTrace LLM calls and monitor performance, costs, and errors via LangSmith.
InnerScopeHearingAutomate AI feature evaluation for PII leakage and hallucination detection in CI pipelines.
