auditing-inference-latency-budget

What problem does it solve?

This Skill diagnoses why a real-time ML inference service is breaching its latency SLO, with special focus on tail behavior that hides behind a healthy mean or P50.

Core Features & Use Cases

• Per-stage latency decomposition: Breaks end-to-end request time into tokenize, feature lookup, preprocess, model forward, postprocess, serialization, and network contributors.
• Tail-driver isolation: Compares slow requests against fast requests to identify the dominant cause of P95/P99 breaches, including input-shape variance, cold-start effects, contention, cache misses, and external dependencies.
• SLO and capacity guidance: Evaluates measured latency against the stated contract and ranks optimizations by expected P99 reduction, implementation effort, and quality risk.
• Use case: A fraud-scoring API has a healthy median but a blown P99; this Skill helps determine whether the bottleneck is batch size, feature lookup, model inference, or infrastructure contention.

Quick Start

Use the auditing-inference-latency-budget skill to review your real-time inference latency logs and identify the dominant P99 tail driver.