llm-extraction-pipeline

What problem does it solve?

LLM-based information extraction often returns malformed or truncated JSON, causing fragile pipelines and wasted time when building structured datasets.

Core Features & Use Cases

• Schema-driven prompt construction: Generates prompts from a structured schema to enforce consistent output shape and field semantics.
• Robust 5-stage JSON parsing: Recovers data from markdown fences, prose-wrapped JSON, and truncated responses using fence stripping, raw JSON import, bracket-block extraction, and partial array recovery.
• JSON-safe normalization for Mathematica types: Sanitizes unsupported values (e.g., Missing, Automatic, DateObject) into JSON-compatible representations before writing JSONL.
• Verbose diagnostics for LLM calls: Provides timing and response-length visibility to quickly distinguish timeouts, early truncation, and parse failures.
• Claim post-processing hooks: Supports normalization/validation steps such as required-field checks, ID generation, and content hashing before persisting records.

Quick Start

Use the llm-extraction-pipeline skill to extract a list of claims from your source text by supplying a schema and requesting JSON-only output from the LLM.