llm-cost-optimization

Start with the three highest-ROI levers: route routine traffic to cheaper models, restructure prompts for a stable cached prefix, and use Batch API or PTU reservations where workloads fit. Instrument per-request token usage first so savings are measurable.

Azure caching is automatic with roughly 50% off input tokens once the first 1,024 tokens form an identical prefix, accruing hits every 128 tokens after. Place stable content first and dynamic content last, since any byte drift before the cache boundary invalidates the prefix.

PTU generally wins above roughly 50% sustained utilization and 150-200M tokens per month on GPT-4o-class models. Deploy first, gather 30-60 days of telemetry, then buy reservations since they guarantee discount, not capacity.

Research shows extractive compression, which selects whole sentences, often outperforms token-pruning methods and enables up to 10x compression with minimal accuracy loss. Use perplexity-based pruning like LLMLingua mainly for query-aware long-context RAG scenarios.

The most common cause is dynamic content placed before static content, such as a timestamp in the system prompt, which invalidates the cached prefix. Verify hits via the cached_tokens field in the API usage object.

No. Microsoft for Startups credits cover only models sold directly by Azure, such as Azure OpenAI. Third-party Marketplace models like Anthropic Claude or Meta Llama are billed separately, so filter the catalog by Direct from Azure.