skills.rest
langwatch
langwatchOFFICIAL
Explore all skills by this author →
GitHub

agent-improve

Turns production traces into tested agent improvements via hypotheses, scenario tests, and reviewable PRs.

3.5k|362|Updated Sep 9, 2023
Search Tags:#hypothesis testing#LLM evaluation#Prompt Engineering#scenario testing#agent improvement#production traces#langwatch
One-click install
npx skills add https://github.com/langwatch/langwatch --skill agent-improve
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-improve
Source: https://github.com/langwatch/langwatch/tree/main/skills/_compiled/native/agent-improve
Command: npx skills add https://github.com/langwatch/langwatch --skill agent-improve

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Teams running LLM agents in production often don't know what to fix next. This Skill mines real production traces and analytics for failure patterns, forms explicit evidence-backed hypotheses, and executes the chosen ones as scenario tests, prompt or code changes in reviewable PRs, evaluators, monitors, and experiments.

Core Features & Use Cases

  • Evidence-driven diagnosis: Queries LangWatch analytics and exports traces to find failure clusters, cost concentration, and quality trends before proposing anything.
  • Hypothesis workflow: Presents 2-4 ranked hypotheses with observation, suspected cause, proposed test, fix, and expected metric impact, then waits for user approval before executing.
  • Concrete execution: Reproduces real failures as scenario tests (with sanitized inputs), ships prompt/code fixes as PRs, adds evaluators and monitors, and runs experiments to settle competing approaches.
  • Use Case: Your support agent mishandles date-sensitive questions in production. The Skill finds the failing traces, hypothesizes stale retrieval, writes a scenario test reproducing the failure, fixes the retrieval code on a branch, and opens a PR linking the evidence.

Quick Start

Ask the agent to analyze my LangWatch production traces and tell me what I should do next to improve my agent.

Frequently Asked Questions about agent-improve

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I improve my AI agent using production traces?▼

Use the langwatch CLI to export traces and query analytics for failure clusters, then form hypotheses ranked by impact. Each approved hypothesis becomes a scenario test reproducing the failure plus a prompt or code fix delivered as a reviewable PR.

How do I turn failing production traces into regression tests?▼

Fetch the exact failing inputs with langwatch trace get, then write scenario tests that reproduce the failure's structure. Sanitize sensitive values with realistic stand-ins and reference the original trace by link instead of pasting raw customer content.

What are the requirements to use the LangWatch agent improvement workflow?▼

You need the langwatch CLI installed, a valid LANGWATCH_API_KEY, and a LangWatch project containing production traces. Without existing traces there is no evidence to mine, so you must first instrument your agent or use dataset-based evaluation instead.

Can I use this if my agent has no production traffic yet?▼

No. Without traces there is no production evidence to mine, so the skill switches method: measure answers against a dataset using prompt optimization or experiments instead, then return once real traffic exists.

What happens when I hit LangWatch free plan limits?▼

The API returns a message stating the free plan limit with an upgrade link. Work within the limits by creating fewer, higher-value resources, and never delete or repurpose existing resources to evade the limit.

Related Skills

Explore More Software Engineering Skills→
mguinadamguinada

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry instrumentation.

Community
Advanced
AlejandroFiginiAlejandroFigini

benchmark

Measure and compare web-page performance metrics like TTFB, FCP, and LCP.

Community
Advanced
sirnsirn

code-review

Run parallel code review agents and generate a unified findings report.

Community
Advanced
lucaslessachaveslucaslessachaves
1

databricks-mlflow-evaluation

Automate MLflow GenAI agent evaluation with scoring, MemAlign judge alignment, and GEPA prompt optimization.

Community
Advanced