speed-aware-llm-pipeline

Implements model routing, budget tracking, retry logic, and prompt caching for LLM API pipelines.

Updated May 2, 2024
One-click install
npx skills add https://github.com/cman131/EatSomethingSourWhenYoureTired --skill speed-aware-llm-pipeline-cman131
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: speed-aware-llm-pipeline
Source: https://github.com/cman131/EatSomethingSourWhenYoureTired/tree/main/.claude/skills/speed-aware-llm-pipeline
Command: npx skills add https://github.com/cman131/EatSomethingSourWhenYoureTired --skill speed-aware-llm-pipeline-cman131

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? LLM API costs grow quickly when every request uses the most expensive model, retries fire on permanent errors, and long system prompts are resent uncached. This Skill provides composable Python patterns to control spend while preserving quality on complex tasks. ## Core Features & Use Cases - Model Routing by Complexity: Automatically selects cheaper models (e.g., Haiku) for simple inputs and reserves expensive models (e.g., Sonnet) for large texts or high item counts. - Immutable Cost Tracking: Tracks cumulative spend with frozen dataclasses and enforces pre-flight budget checks before each API call. - Narrow Retry Logic and Prompt Caching: Retries only transient errors (rate limits, connection, server errors) with exponential backoff, and caches long system prompts via cache_control. - Use Case: When batch-processing hundreds of documents through the Claude API, route each item to the right model tier, reject requests that would exceed a $1.00 budget, and log per-interaction costs to a JSONL metrics file via the included PostToolUse hook. ## Quick Start Ask the AI to build a cost-aware LLM pipeline that routes requests between Haiku and Sonnet based on input size, enforces a budget limit, and caches the system prompt.

Frequently Asked Questions about speed-aware-llm-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce LLM API costs in Python?▼

Route requests by task complexity so simple inputs use cheaper models like Haiku while complex ones use Sonnet. Combine this with prompt caching for long system prompts and pre-flight budget checks that reject requests before spending.

How to choose between Claude Haiku and Sonnet models?▼

Select the model based on input size and item count thresholds, for example routing to Sonnet when text exceeds 10,000 characters or 30 items. Haiku costs roughly 3-4x less, making it the default for simple tasks.

Which API errors should be retried for LLM calls?▼

Retry only transient errors: APIConnectionError, RateLimitError, and InternalServerError, using exponential backoff. Authentication and bad request errors are permanent and should fail immediately to avoid wasting budget.

Does prompt caching work with the Anthropic API?▼

Yes, the Anthropic messages API supports ephemeral prompt caching by adding cache_control to a system block. Cache system prompts over 1024 tokens to save both cost and latency on repeated requests.

How do I track LLM spending per session?▼

Use an immutable CostTracker dataclass that accumulates CostRecord entries per API call and compares total cost against a budget limit. The included PostToolUse hook also appends per-interaction cost estimates to a JSONL metrics file.