eval-check

Agent quality evaluations run automatically after code modifications by executing deterministic checks first, then conditionally running LLM-based tests for groundedness, correctness, trajectory, citations, and disclaimers.

Tiered evaluation execution runs fast, deterministic tests first, then prompts for confirmation before executing expensive LLM-based evaluations. This mechanism optimizes testing resources by validating basic correctness before committing to costly LLM inference checks.

Yes, selective evaluations allow running only relevant LLM tests based on the type of code changes detected. The skill categorizes modifications in core agent packages and selects matching evaluations to verify groundedness, trajectory, and citations.

Yes, pnpm is required as a dependency to automate the execution of agent quality evaluations triggered by code modifications. This package manager orchestrates the scripts that run deterministic and LLM-based tests to prevent agent performance regressions.

Automated code change detection identifies modified agent files across core, tools, and services. It categorizes these changes and selects relevant evaluations, running deterministic checks before conditionally executing LLM-based tests to prevent regressions.

LLM-based agent evaluations are expensive and require confirmation before execution. The skill mitigates this by running fast deterministic tests first and only conditionally running LLM tests for groundedness, correctness, trajectory, citations, and disclaimers based on detected changes.