skills.rest
GoogleCloudPlatform
GoogleCloudPlatformOFFICIAL
Explore all skills by this author →
WebsiteGitHub

quality-flywheel

Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.

783|307|Updated May 27, 2021
Search Tags:#synthetic data#quality improvement#agent testing#LLM metrics#vertex ai#genai evaluation#RAG evaluation
One-click install
npx skills add https://github.com/GoogleCloudPlatform/vertex-ai-samples --skill quality-flywheel
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quality-flywheel
Source: https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/skills/quality-flywheel
Command: npx skills add https://github.com/GoogleCloudPlatform/vertex-ai-samples --skill quality-flywheel

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vertexai, google-genai, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Evaluating GenAI models and agents is hard: teams often lack test data, don't know which metrics to use, and struggle to turn low scores into concrete fixes. This Skill provides an end-to-end evaluation workflow using the Google GenAI Evaluation SDK, from dataset creation through iterative quality improvement.

Core Features & Use Cases

  • Dataset Creation: Build evaluation datasets from ADK session traces or generate synthetic user scenarios for cold-start situations, using canonical types like EvalCase, AgentData, and ConversationTurn.
  • Metric Selection & Customization: Choose from predefined AutoRater metrics (hallucination_v1, tool_use_quality_v1, multi_turn_task_success_v1), computation metrics (exact_match, rouge_l_sum), or write custom LLMMetric and CodeExecutionMetric judges.
  • Result Analysis & Iteration: Execute evals via client.evals.evaluate(), analyze rubric verdicts and failure patterns, and apply targeted prompt or agent-code fixes in a measurable improvement loop.
  • Use Case: A RAG agent scores 0.45 on hallucination. The Skill identifies the grounding failure, suggests system-prompt fixes from its failure-pattern reference, and re-runs the eval to verify improvement.

Quick Start

Ask the assistant to evaluate your agent or model with the Vertex Evaluation SDK, providing your GCP project ID and location when prompted.

Frequently Asked Questions about quality-flywheel

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a GenAI agent with the Vertex AI Evaluation SDK?▼

Build an EvaluationDataset of EvalCase objects containing AgentData conversation turns, then call client.evals.evaluate() with metrics like multi_turn_trajectory_quality_v1 or tool_use_quality_v1. Initialize the client with your GCP project ID and location first.

How do I create evaluation data when I have no test cases?▼

Use client.evals.generate_user_scenarios with a UserScenarioGenerationConfig specifying scenario count, simulation instruction, and environment data. Then run client.evals.run_inference with a user simulator to produce a populated multi-turn dataset.

Which metrics should I use for a tool-calling agent?▼

Use tool_use_quality_v1 and multi_turn_task_success_v1 for LLM-judged quality, plus computation metrics tool_call_valid, tool_name_match, and tool_parameter_kv_match for deterministic checks against reference tool calls.

How do I convert ADK session logs into an evaluation dataset?▼

Run the parse_adk_traces.py script with --input pointing to your ADK session JSON. It segments events into ConversationTurns, extracts agent configs, and outputs an EvaluationDataset JSON ready for client.evals.evaluate().

Can I write a custom LLM-as-a-judge metric in Vertex AI?▼

Yes. Create a types.LLMMetric with a prompt_template referencing fields like {prompt} and {response}, or use MetricPromptBuilder for structured criteria and rating scales. For deterministic checks, use CodeExecutionMetric with a custom evaluate function.

Why is my hallucination or grounding score low and how do I fix it?▼

Low hallucination_v1 or grounding_v1 scores usually mean the system prompt lacks grounding instructions or retrieved context is not injected into the prompt. Add explicit instructions to answer only from provided context, lower temperature, and re-run the eval to verify.

Related Skills

Explore More Software Engineering Skills→
auxi-wardrobeauxi-wardrobe
1

ck:google-adk-python

Develop and deploy Python AI agents using Google's Agent Development Kit (ADK).

Official
Advanced
beam-ai-teambeam-ai-team

beam-ape-optimizer

Automate iterative prompt optimization for Beam agent nodes using APE.

Official
Advanced
Supporter09Supporter09

langsmith-observability

Trace, evaluate, and monitor LLM application runs with LangSmith.

Community
Advanced
cuba6112cuba6112

eval-frameworks

Provides evaluation frameworks for LLM outputs in RAG systems.

Community
Intermediate