trace-annotation-tool

Name: trace-annotation-tool
Availability: InStock
Author: maragu

Generate a FastHTML web app for annotating LLM traces with pass/fail/defer status.

10|Updated Feb 19, 2026

Search Tags:#llm #error analysis #annotation #trace analysis #open coding #web app generation

Ready to run

npx skills add https://github.com/maragudk/evals-skills --skill trace-annotation-tool

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: trace-annotation-tool
Source: https://github.com/maragudk/evals-skills/tree/main/skills/trace-annotation-tool
Command: npx skills add https://github.com/maragudk/evals-skills --skill trace-annotation-tool

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the creation of a local web application for qualitative analysis of LLM traces, enabling users to perform open coding and identify failure modes efficiently.

Core Features & Use Cases

Custom Web App Generation: Creates a self-contained Python web app using FastHTML, TailwindCSS, and HTMX.
Trace Annotation: Allows users to review LLM traces, add freeform notes, and mark traces as pass/fail/defer.
Use Case: When you need to manually inspect and label the outputs of an LLM pipeline to understand its errors before building a structured failure taxonomy.

Quick Start

Use the trace-annotation-tool skill to generate a local web application for annotating LLM traces.

Frequently Asked Questions about trace-annotation-tool

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build a custom web app for LLM trace annotation and error analysis?▼

You can generate a custom Python web app for LLM trace annotation using FastHTML, TailwindCSS, and HTMX. This tool automates the creation of a local interface to review traces, add freeform notes, and mark pass/fail/defer statuses.

What is the best way to perform open coding on LLM traces to identify failure modes?▼

Performing open coding on LLM traces is best handled by a custom annotation web app. It allows you to manually inspect outputs, apply freeform notes, and assign pass/fail/defer labels to systematically build a failure taxonomy.

Does this trace annotation tool support keyboard shortcuts for reviewing LLM pipelines?▼

Yes, the generated trace annotation tool supports keyboard shortcuts for reviewing LLM pipelines. This allows you to efficiently navigate structured data input and quickly annotate traces within the FastHTML web interface.

Can I use FastHTML and HTMX to create a local interface for qualitative error analysis?▼

Yes, you can use FastHTML and HTMX to create a local web application for qualitative error analysis. The generated interface is self-contained, responsive, and specifically designed for open coding LLM traces.

What do I need to generate a self-contained Python web app for annotating LLM outputs?▼

To generate a self-contained Python web app for annotating LLM outputs, you need structured trace data input. The Skill uses FastHTML, TailwindCSS, and HTMX to build the local annotation interface without external dependencies.

When should I use a dedicated trace annotation tool instead of manual LLM error tracking?▼

You should use a dedicated trace annotation tool when you need to manually inspect large volumes of LLM outputs to understand errors before building a structured failure taxonomy. It provides structured data input and pass/fail/defer tracking.