inference-performance

What problem does it solve?

This Skill helps diagnose and improve the performance of LLM inference in production environments, addressing issues like TTFT/TPOT/throughput, batching strategy, GPU sizing, and debugging low GPU utilization.

Core Features & Use Cases

• Performance Diagnostics: Identify bottlenecks in LLM inference, such as prefill vs decode phase issues, TTFT spikes, and OOM errors.
• Batching Strategies: Choose optimal batching strategies for LLM inference, considering prefill and decode phases.
• GPU Sizing: Determine appropriate GPU sizes for serving LLMs based on performance metrics.
• Use Case: For a company deploying a new LLM service, this Skill can help ensure that the service meets its latency and throughput requirements by optimizing the configuration and batching strategy.

Quick Start

Run the inference-performance skill to analyze the performance of your LLM inference service and optimize it for better throughput and lower latency.