error-analysis

Analyze LLM pipeline traces to categorize failure modes and compute failure rates.

5|Updated Oct 22, 2025

Search Tags:#debugging #llm #evaluation #pipeline #error analysis #failure modes

Ready to run

npx skills add https://github.com/marchatton/agent-skills --skill error-analysis-marchatton

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: error-analysis
Source: https://github.com/marchatton/agent-skills/tree/main/.agents/skills/08-evals/error-analysis
Command: npx skills add https://github.com/marchatton/agent-skills --skill error-analysis-marchatton

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps users systematically identify, categorize, and quantify failure modes within an LLM pipeline, enabling targeted improvements.

Core Features & Use Cases

Systematic Failure Identification: Guides users through analyzing LLM traces to pinpoint root causes of errors.
Categorization and Labeling: Facilitates grouping similar failures and labeling all traces against these categories.
Prioritization: Computes failure rates to help users focus on the most impactful issues.
Use Case: After a significant update to an LLM-powered customer support bot, use this Skill to analyze recent conversations, identify why certain queries are failing, and prioritize fixes for the most common issues.

Quick Start

Use the error-analysis skill to help categorize the failure modes observed in the provided LLM traces.

Frequently Asked Questions about error-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I identify and categorize failure modes in an LLM pipeline?▼

To identify and categorize LLM pipeline failure modes, collect execution traces, note specific failures, group them into actionable categories, label all traces against these groups, and compute failure rates to prioritize fixes.

What is the best way to analyze LLM traces after a metric drop?▼

Analyzing LLM traces after a metric drop involves systematically reviewing recent logs, categorizing observed errors into distinct failure modes, and computing their frequencies to pinpoint the root cause for prioritized incident response.

Can I use error analysis for post-change evaluation of an LLM-powered application?▼

Yes, post-change analysis is a core use case. You can evaluate an updated LLM pipeline by analyzing recent conversation traces, grouping the failures into actionable categories, and quantifying error rates to target improvements.

How does analyzing failure rates help with debugging an LLM evaluation pipeline?▼

Computing failure rates during LLM debugging helps prioritize issues by quantifying how often each categorized failure mode occurs. This allows you to focus debugging efforts on the most impactful pipeline errors first.

When should I perform systematic error analysis on my LLM traces?▼

Perform systematic error analysis when starting new evaluation projects, investigating metric drops, responding to incidents, or conducting post-change analysis. It requires collecting traces and grouping them into actionable categories to prioritize fixes.