error-analysis

Analyze LLM pipeline traces to categorize and prioritize failure modes.

1.6k|164|Updated Mar 1, 2026

Search Tags:#debugging #llm #evaluation #failure analysis #categorization #trace analysis

Ready to run

npx skills add https://github.com/hamelsmu/evals-skills --skill error-analysis-hamelsmu

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: error-analysis
Source: https://github.com/hamelsmu/evals-skills/tree/main/skills/error-analysis
Command: npx skills add https://github.com/hamelsmu/evals-skills --skill error-analysis-hamelsmu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps users systematically identify and categorize failure modes in an LLM pipeline by guiding them through reading traces and building a catalog of how the system fails.

Core Features & Use Cases

Trace Analysis: Guides users through reviewing LLM traces (inputs, intermediate calls, tool uses, outputs).
Failure Categorization: Assists in grouping similar failures into distinct, actionable categories.
Prioritization: Helps compute failure rates to identify areas needing the most attention.
Use Case: After a significant LLM pipeline change, use this skill to understand new failure modes and prioritize fixes by analyzing ~100 traces.

Quick Start

Use the error-analysis skill to help me categorize failures by reading through the provided traces.

Frequently Asked Questions about error-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I categorize LLM pipeline failure modes from trace analysis?▼

Trace analysis for LLM failures involves reviewing inputs, intermediate calls, tool uses, and outputs to identify where the pipeline breaks. It systematically guides you through reading traces to build a catalog of how your LLM system fails.

What is the best way to prioritize fixes after significant LLM pipeline changes?▼

Prioritize fixes after pipeline changes by analyzing around 100 traces to compute failure rates. This identifies the most frequent failure modes, guiding your debugging efforts toward the highest-impact areas requiring immediate attention.

How do I systematically debug LLM evaluation failures in production?▼

Systematically debug LLM evaluation failures by collecting production traces and applying user-driven annotation. This administrative workflow supports incident response by helping you identify, group, and categorize failure modes to address production metric drops.

How many traces do I need to analyze to identify new failure modes?▼

Analyze approximately 100 traces to identify new failure modes. This sample size provides sufficient data for LLM-assisted clustering and failure rate computation, ensuring accurate categorization and prioritization of issues after significant pipeline changes.

Can I use LLM-assisted clustering to refine failure categories?▼

Yes, you can use LLM-assisted clustering to refine failure categories. This technique groups similar failures into distinct, actionable categories, enhancing the annotation process when building a comprehensive catalog of LLM pipeline issues.