context-optimization

To optimize LLM context, you can apply compaction, observation masking, KV-cache tuning, and partitioning. These techniques reduce token waste by summarizing near-limit text, replacing verbose tool outputs, and maximizing cache reuse for lower latency.

Observation masking manages token budgets by replacing verbose tool outputs with compact references. This technique keeps the active LLM context within its token limits while preserving essential information from previous interactions.

Context partitioning handles large documents by distributing the workload across sub-agents with isolated contexts. This technique divides token consumption so the primary agent avoids hitting its context window limit.

You can improve latency by arranging stable prompts to maximize KV-cache reuse. Keeping the prompt structure consistent allows the model to retrieve cached key-value pairs instead of recalculating them for long conversations.

You should apply compaction when a conversational agent approaches its token limit while processing large documents. This technique summarizes near-limit context to retain essential information and keep cost and latency in check.