llm-eval-grounded-theory

What problem does it solve?

This Skill addresses the challenge of building and maintaining trustworthy LLM-as-judge pipelines, ensuring accurate evaluation and continuous monitoring of AI systems.

Core Features & Use Cases

• Qualitative-to-Quantitative Evaluation: Automates the process of moving from qualitative error analysis to quantitative evaluation, including open coding, axial coding, and rubric design.
• LLM Calibration: Provides a framework for calibrating LLM judges against gold sets and monitoring their performance in production.
• Continuous Monitoring: Offers tools for ongoing evaluation and monitoring of LLM systems, including offline regression and online drift detection.

Quick Start

Execute the skill to begin the LLM evaluation pipeline by collecting trace data and proceeding through the stages outlined in the documentation.