skills.rest
langwatch
langwatchOFFICIAL
Explore all skills by this author →
GitHub

eval-triage

Diagnose failing LangWatch experiment runs down to failing rows, evaluators, and root causes.

3.5k|362|Updated Sep 9, 2023
Search Tags:#root cause#evaluation#LLM evaluation#Debugging#Evaluator#langwatch#experiment triage
One-click install
npx skills add https://github.com/langwatch/langwatch --skill eval-triage
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-triage
Source: https://github.com/langwatch/langwatch/tree/main/skills/_compiled/native/eval-triage
Command: npx skills add https://github.com/langwatch/langwatch --skill eval-triage

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When an LLM experiment or evaluation run fails or scores regress, it is hard to tell whether the agent, the evaluator, the dataset, or the judge model is at fault. This Skill provides a structured triage workflow that isolates the exact failing rows and evaluators, then guides you to a root cause.

Core Features & Use Cases

  • Failing Run Discovery: Locate failing experiment runs and distinguish execution failures from score regressions using the LangWatch CLI.
  • Row and Evaluator Isolation: Filter results to failing rows and per-evaluator breakdowns to determine whether a regression is concentrated in one scorer or spread across all of them.
  • Root Cause and Prevention: Compare against the last passing run, inspect production traces and datasets, then add evaluators or monitors to prevent recurrence.
  • Use Case: An experiment run completes but accuracy drops 15%. Use this Skill to filter the failed rows, discover the regression is isolated to one LLM-judge evaluator whose model changed, and fix the judge configuration before re-running.

Quick Start

Triage my failing LangWatch experiment run and find which rows and evaluators caused the score regression.

Frequently Asked Questions about eval-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find why a LangWatch experiment run failed?▼

Use langwatch experiment status to check whether the run errored out or completed with regressed scores. Execution failures show the error directly, while score regressions require filtering failing rows with langwatch experiment results --filter failed.

How do I isolate failing rows in an evaluation run?▼

Run langwatch experiment results <slug> --filter failed to keep only rows that failed at least one evaluator. Add --evaluator <name> to see one evaluator's column and determine whether the regression is concentrated in a single scorer.

Can an LLM judge evaluator cause false score regressions?▼

Yes. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings unrelated to the agent. Check the model in the evaluator settings and examine the score distribution rather than just pass/fail thresholds.

What do I need to run LangWatch experiment triage?▼

You need the langwatch CLI installed and a valid LANGWATCH_API_KEY configured. The workflow works with any coding agent and uses commands like experiment list, status, results, and evaluator get.

How do I prevent an evaluation failure from recurring?▼

If the failure mode was not covered by any evaluator, create one with langwatch evaluator create and wire it into the experiment. If it only appears in production, set up a monitor with langwatch monitor create for online evaluation.

Related Skills

Explore More Software Engineering Skills→
mguinadamguinada

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry instrumentation.

Community
Advanced
tadod12tadod12

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry.

Community
Advanced
alessandro9110alessandro9110
4

analyzing-mlflow-trace

Parse MLflow trace JSON to diagnose execution failures and assess quality.

Community
Advanced
Arize-aiArize-ai
11.0k

phoenix-cli

Fetch traces, inspect spans, and review experiments via the Phoenix CLI.

Official
Intermediate