error-analysis

Categorize failure modes in LLM pipelines by analyzing trace data.

Updated May 5, 2026

Search Tags:#debugging #llm #pipeline #trace #error analysis #evals

Ready to run

npx skills add https://github.com/iani-kuli/harness_bro --skill error-analysis-iani-kuli

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: error-analysis
Source: https://github.com/iani-kuli/harness_bro/tree/main/.claude/skills/curated/evals/error-analysis
Command: npx skills add https://github.com/iani-kuli/harness_bro --skill error-analysis-iani-kuli

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill addresses the challenge of debugging LLM pipelines by providing a structured, repeatable framework to identify, categorize, and prioritize failure modes in model outputs.

Core Features & Use Cases

Trace Collection & Review: Guides the user through sampling and annotating representative pipeline traces to identify root causes.
Failure Categorization: Facilitates the emergence of actionable failure categories, helping teams distinguish between prompt issues, tool misconfigurations, and engineering bugs.
Prioritization Framework: Provides a clear methodology to compute failure rates and decide which issues warrant custom evaluators versus simple fixes.

Quick Start

Use the error-analysis skill to begin reviewing a batch of 100 traces and cataloging the primary failure modes in my current pipeline.

Frequently Asked Questions about error-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I systematically identify and categorize LLM pipeline failure modes?▼

To systematically identify LLM pipeline failure modes, manually sample and review trace data including inputs, intermediate steps, and outputs to generate actionable failure categories and prioritize remediation efforts.

What is the best way to debug LLM pipelines using trace data?▼

The best way to debug LLM pipelines using trace data is to sample representative traces and annotate them to distinguish between prompt issues, tool misconfigurations, and engineering bugs.

Can I use this framework to decide which LLM failures need custom evaluators?▼

Yes, you can use this framework to compute failure rates from your trace data, which provides a clear methodology to decide which issues warrant custom evaluators versus simple fixes.

When do I need a structured framework for LLM error analysis?▼

You need a structured framework for LLM error analysis during iterative development cycles, production incident response, and quality assurance for AI-driven applications to ensure repeatable debugging.

Does error analysis require manual review of LLM traces?▼

Yes, error analysis requires manual review of trace inputs, intermediate steps, and outputs to accurately generate actionable failure categories and prioritize remediation efforts.