skills.rest
grafana
grafanaOFFICIAL
Explore all skills by this author →
Website𝕏@grafanaGitHub

agento11y-test-starter

Generates offline test suites and runner stubs for AI agents before production deployment.

581|46|Updated Mar 23, 2026
Search Tags:#Test Generation#prompt testing#agent evaluation#llm judge#grafana cloud#Observability#offline experiments
One-click install
npx skills add https://github.com/grafana/gcx --skill agento11y-test-starter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agento11y-test-starter
Source: https://github.com/grafana/gcx/tree/main/claude-plugin/skills/agento11y-test-starter
Command: npx skills add https://github.com/grafana/gcx --skill agento11y-test-starter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires agento11y, python-dotenv.

What problem does it solve?

Developers shipping AI agents often have no test cases and no scoring strategy before real traffic exists, making pre-production quality evaluation impossible. This Skill reads the agent's actual code and produces a grounded starter test suite plus a runnable offline experiment harness.

Core Features & Use Cases

  • Code-Grounded Test Generation: Reads the agent's entrypoint, system prompt, and tool definitions to write a labeled suite YAML with happy, edge, and adversarial cases citing file:line references.
  • Evaluator Recommendation: Picks 3-6 evaluators (llm_judge or deterministic kinds) mapped to what the agent actually does, reusing the agent's existing LLM provider for judging.
  • Runner Bootstrap: Generates a run_experiment.py stub wired to the Agent Observability experiments SDK with two clearly marked holes to fill, adapting to the agent's runnability (easy, in-process, or full-stack) and language (Python/Go native, subprocess bridge otherwise).
  • Use Case: Before shipping a new support chatbot agent, ask the Skill to analyze its code, produce evals/support-bot-starter.yaml with adversarial prompt-injection cases, and optionally run a smoke experiment against Grafana Cloud.

Quick Start

Ask the assistant to write test cases for your agent before shipping and set up an offline evaluation suite for it.

Frequently Asked Questions about agento11y-test-starter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test my AI agent before shipping it to production?▼

Run this Skill against your agent's repository to generate a starter suite YAML with happy, edge, and adversarial cases grounded in your actual code, plus a run_experiment.py runner stub. Fill the two marked holes (run_agent and the judge model call), then run the offline experiment.

How do I write test cases for an LLM agent with no traffic?▼

The Skill reads your agent's entrypoint, system prompt, and tool definitions, then derives at least six labeled cases weighted toward edge and adversarial scenarios. Cases are drafts to review and extend, never presented as validated.

Does the Agent Observability experiments SDK support TypeScript or Java agents?▼

No, the experiments SDK exists only for Python and Go. For TypeScript, Java, or .NET agents, the Skill delivers the language-neutral recommendations and YAML, and offers a Python runner that shells out to the agent across a process boundary as a labeled option.

What credentials are needed to run an offline agent evaluation?▼

You need AGENTO11Y_ENDPOINT and AGENTO11Y_AUTH_TOKEN from the Grafana Cloud plugin Connection page, plus the agent's own LLM provider API key. The Skill never mints tokens and only reads them from your environment or a gitignored .env file.

What is the difference between offline test suites and online eval rules?▼

This Skill only creates offline experiments that publish one run's scores and never creates tenant-level evaluators, rules, or guards. Online evaluation of live traffic via Agent Observability Rules or guard hooks is handled by the separate agento11y-prod-setup skill after deployment.

Why does my experiment run fail with a 401 owned by another actor error?▼

This happens when the experiment ID is reused across different auth actors or when run and trials use different actors. Use a fresh timestamped experiment_id per run and set a stable AGENTO11Y_INGEST_ACTOR value.

Related Skills

Explore More Software Engineering Skills→
hackefellerhackefeller
1

dspy-ruby

Build type-safe LLM applications in Ruby with DSPy.rb.

Official
Advanced
langwatchlangwatch
3.5k

online-evaluations

Configure LangWatch online evaluations and guardrails for production LLM traffic.

Official
Intermediate
davduncdavdunc

Evals

Run regression and capability evaluations for multi-turn AI agent workflows.

Community
Advanced
DigiStrique-SolutionsDigiStrique-Solutions
1

eval-harness

Automate AI prompt and output evaluation against ground-truth fixtures.

Official
Advanced