continuous-eval

Automate post-deployment quality assessment for AI agents by running Ragas-based metrics against golden datasets. This Skill performs regression gate checks immediately after deployment and at hourly intervals to ensure continuous reliability.

Ragas metrics for AI agent safety and performance include faithfulness, answer relevance, context precision, toxicity, and PII leakage. These metrics evaluate both output quality and safety risks during continuous assessment.

Set up continuous evaluation for AI agents by running the Skill in a Python runtime with the Ragas library installed. It executes in Claude Code or Kiro environments to evaluate latest deployment versions against golden datasets.

Run regression gate checks on AI agents post-deployment using managed golden datasets. The Skill compares new deployment outputs against established criteria to pass or fail the regression gate automatically.

Detect PII leakage and toxicity in AI responses using integrated Presidio and Detoxify libraries. These dependencies enable the Skill to assess safety risks alongside standard Ragas quality metrics during post-deployment evaluation.

Monitor AI agent performance continuously by running scheduled evaluations at hourly intervals post-deployment. The Skill uses Langfuse for tracing and Ragas metrics to provide ongoing feedback on agent reliability.