skills.rest
google-gemini
google-geminiOFFICIAL
Explore all skills by this author →
Website𝕏@googleaidevsGitHub

behavioral-evals

Create, run, fix, and promote behavioral evaluations for agent decision logic.

107k|14.5k|Updated Apr 17, 2025
Search Tags:#Vitest#prompt-engineering#regression-testing#agent-testing#Gemini-CLI#behavioral-evals#trajectory-auditing
One-click install
npx skills add https://github.com/google-gemini/gemini-cli --skill behavioral-evals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: behavioral-evals
Source: https://github.com/google-gemini/gemini-cli/tree/main/.gemini/skills/behavioral-evals
Command: npx skills add https://github.com/google-gemini/gemini-cli --skill behavioral-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vitest, and includes references (resource) and assets (resource) components.

What problem does it solve?

Validating changes to prompts, tools, or agent architecture is difficult because standard integration tests verify functionality, not the agent's underlying decision-making. This Skill provides a structured workflow to write, debug, and stabilize behavioral evaluations that audit tool trajectories and prevent regressions in agent steerability.

Core Features & Use Cases

  • Rig Selection: Choose between evalTest (subprocess) for standard workspace tests and appEvalTest (in-process) for UI or breakpoint-heavy scenarios.
  • Assertion Strategies: Audit agent decisions using breakpoints, tool log inspection, and mock MCP facades to verify efficiency and correctness.
  • Failure Investigation & Fixing: Diagnose failing evals by reading trajectory logs, adjusting system prompts in snippets.ts, and verifying stability across multiple Gemini models.
  • Promotion Workflow: Incubate new tests under the USUALLY_PASSES policy and promote them to ALWAYS_PASSES after achieving 100% stability across nightly runs.

Quick Start

Use the behavioral-evals skill to create a new USUALLY_PASSES eval test that verifies the agent reads a file before writing to it.

Frequently Asked Questions about behavioral-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a behavioral evaluation for an AI agent?▼

Use `evalTest` from `./test-helper.js` for standard subprocess tests or `appEvalTest` from `./app-test-helper.js` for UI tests. Seed the workspace with realistic files, write assertions on `rig.readToolLogs()`, and start new tests with the `USUALLY_PASSES` policy.

What is the difference between evalTest and appEvalTest?▼

`evalTest` runs the CLI in a separate subprocess and waits for exit, making it safe for trajectory auditing but incompatible with breakpoints. `appEvalTest` runs in-process, allowing `setBreakpoint` triggers for interactive prompts and Ink rendering tests.

Why does my behavioral eval test hang indefinitely?▼

Breakpoints (`setBreakpoint`) pause execution, which causes `evalTest` to hang because `rig.run()` waits for process exit before assertions run. Use `appEvalTest` for breakpoints, or audit tool logs instead for standard trajectory tests.

How do I promote a USUALLY_PASSES eval to ALWAYS_PASSES?▼

Audit nightly logs from `evals-nightly.yml` to confirm the test passed 100% of the time across all enabled models for the last 7 runs. Then change the policy argument in the test file from `USUALLY_PASSES` to `ALWAYS_PASSES`.

Can I fix a failing eval by changing the test prompt?▼

Changing the test prompt is a last resort because prompts are often vague by design. The primary fix trigger is adjusting tool descriptions or system prompts in `@packages/core/src/prompts/snippets.ts` to maintain test fidelity.

Related Skills

Explore More Software Engineering Skills→
JannaelJannael
16

Bun

Run, bundle, test, and deploy JavaScript/TypeScript apps with Bun CLI.

Community
Advanced
Negentropy-LabyNegentropy-Laby
3

test-flakiness

Analyze CI logs and test histories to identify intermittent failures.

Official
Advanced
0xDarkMatter0xDarkMatter
29

testing-patterns

Standardize testing strategies across languages with naming, Arrange-Act-Assert, mocking, and fixtures.

Community
Advanced
BagelHoleBagelHole
46

agent-evals

Build automated evaluation suites for AI agents with golden datasets and regression gates.

Community
Advanced