agentic-eval

An evaluator-optimizer loop separates generation and evaluation models to iteratively refine agent outputs. You configure iteration counts and convergence checks to automatically return the best validated output based on multi-dimensional rubric scoring.

LLM-as-judge uses a separate model to evaluate outputs against a structured rubric. Rubric-based scoring provides multi-dimensional, calibrated scoring with actionable feedback to drive concrete improvements and mitigate evaluator bias.

Yes, you can configure separate generator and judge models within the evaluation workflow. This separation mitigates bias by ensuring the model evaluating the output is independent from the model that generated it.

Automated self-critique for code generation applies code-focused testing and reflection patterns. The agent evaluates its own code against configured rubrics and iteratively refines the output until it passes structural and functional validation checks.

Yes, the skill requires the anthropic and pytest Python packages. The anthropic package handles model interactions for generation and judging, while pytest supports the code-focused testing and validation components.

Rubric-based validation depends on the quality and domain-awareness of the configured rubric. If the rubric lacks multi-dimensional calibration or specific domain constraints, the automated evaluation may fail to identify nuanced errors in complex structured outputs.