strands-evals

What problem does it solve?

This Skill provides a comprehensive framework for evaluating AI agents and LLM applications, covering various aspects such as output validation, trajectory analysis, tool usage assessment, interaction evaluation, and automated experiment generation.

Core Features & Use Cases

• Multiple Evaluation Types: Output evaluation, trajectory analysis, tool usage assessment, and interaction evaluation
• Multimodal Evaluation: MLLM-as-a-Judge evaluators for image-to-text tasks with built-in rubrics
• Dynamic Simulators: Multi-turn conversation simulation with realistic user behavior, goal-oriented interactions, and LLM-powered tool simulation with shared state
• LLM-as-a-Judge: Built-in evaluators using language models for sophisticated assessment with structured scoring
• Trace-based Evaluation: Analyze agent behavior through OpenTelemetry execution traces
• Automated Experiment Generation: Generate comprehensive test suites from context descriptions
• Custom Evaluators: Extensible framework for domain-specific evaluation logic
• Experiment Management: Save, load, and version your evaluation experiments with JSON serialization
• Built-in Scoring Tools: Helper functions for exact, in-order, and any-order trajectory matching
• Failure Detection & Root Cause Analysis: Automatically detect failures in agent sessions and diagnose root causes with actionable fix recommendations
• Chaos Testing: Deterministic fault injection via Strands plugin hooks — simulate tool timeouts, network errors, and response corruption to evaluate agent resilience
• Red Team Evaluation: Adversarial safety testing with built-in attack strategies (Crescendo, GOAT, PAIR, BadLikertJudge, SequentialBreak); see src/strands_evals/experimental/redteam/README.md
• Use Case: Imagine you want to evaluate an AI agent's performance in a multi-turn conversation. Use this Skill to simulate realistic user interactions and assess the agent's response quality.

Quick Start

Use the strands-evals skill to run an experiment with the provided cases and evaluators.