agent-platform-eval-flywheel

What problem does it solve?

This Skill solves the challenge of systematically measuring and improving the quality of GenAI models and agents on Google Cloud. Without a structured evaluation workflow, teams struggle to identify failure patterns, track quality improvements, and avoid regressions when iterating on prompts or model configurations.

Core Features & Use Cases

• Quality Flywheel Workflow: Guides users through a five-stage iterative process—prepare data, run inference, grade with metrics, analyze failures, and optimize—to continuously improve AI system quality.
• Multi-Format Evaluation: Supports single-turn model evaluation, multi-turn agent evaluation with tool calls, synthetic data generation for cold starts, and custom metric creation using LLM-as-judge or code-based approaches.
• Production-Grade Tooling: Includes safety tiers for confirmation, failure clustering for large-scale analysis, HTML report generation, and regression detection when comparing evaluation results across iterations.

Quick Start

Use the agent-platform-eval-flywheel skill to evaluate your GenAI agent or model by preparing an evaluation dataset, running inference with the Agent Platform SDK, and grading the results with predefined or custom metrics.