skills.rest
letta-ai
letta-aiOFFICIAL
Explore all skills by this author →
Website𝕏@letta_aiGitHub

llm-inference-batching-scheduler

Optimize LLM inference batching and scheduling under latency and padding constraints.

134|21|Updated Nov 12, 2025
Search Tags:#optimization#llm#latency#scheduling#batching#inference#cost
One-click install
npx skills add https://github.com/letta-ai/skills --skill llm-inference-batching-scheduler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-inference-batching-scheduler
Source: https://github.com/letta-ai/skills/tree/main/ai/benchmarks/letta/terminal-bench-2/trajectory-feedback/llm-inference-batching-scheduler
Command: npx skills add https://github.com/letta-ai/skills --skill llm-inference-batching-scheduler

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill provides guidance for solving LLM inference batching and scheduling optimization problems, where requests must be grouped into batches while minimizing cost, padding waste, and latency.

Problem Understanding

Before implementation, thoroughly analyze the problem structure:

Constraint Analysis

  1. Identify all hard constraints - Extract exact limits for:

    • Maximum unique shapes allowed (e.g., ≤ 8 shapes across all buckets)
    • Latency thresholds (P95, P99)
    • Cost budget thresholds
    • Padding ratio limits

  2. Compute hard bounds early - Before coding, calculate:

    • Minimum possible padding from alignment requirements
    • Minimum number of batches required for coverage
    • Maximum achievable efficiency given constraints

  3. Decompose the cost function - Understand each component:

    • Per-batch overhead (fixed cost per batch)
    • Shape compilation costs (often quadratic in sequence length)
    • Prefill/decode costs (variable per request)
    • Document as: Cost ≈ overhead × num_batches + shape_compile_cost + prefill_cost + decode_cost

Data Analysis

  1. Profile the request distribution - Examine:

    • Distribution of prompt lengths (prompt_len)
    • Distribution of generation lengths (gen_len)
    • Identify outliers that may disproportionately impact metrics

  2. Verify coverage requirements - Ensure:

    • The largest prompt_len in each bucket is covered by chosen shapes
    • Edge cases with extreme gen_len values are handled

Implementation Approach

Build Reusable Evaluation Infrastructure

Before iterating on parameters, create a systematic evaluation harness:

1. Write a function that takes parameters (shape_list, gen_bucket_sizes) and returns all metrics
2. Include automatic constraint verification with assertions
3. Enable rapid parameter comparison without manual re-runs

Parameter Search Strategy

Avoid random trial-and-error. Instead:

  1. Grid search for small parameter spaces - When parameters are bounded (e.g., gen_bucket_size in [15, 50]), systematically evaluate combinations

  2. Binary search for single parameters - When optimizing one parameter while holding others fixed, use binary search to find optimal values

  3. Document the optimization landscape - Track which parameter combinations produce which metric values to understand trade-offs

Shape Selection Guidelines

When selecting shapes for sequence length bucketing:

  1. Analyze the length distribution - Choose shapes that minimize padding for the most common lengths
  2. ** Consider power-of-two or geometric progressions** - These often balance coverage vs. shape count
  3. Account for both buckets jointly - If shapes are shared across buckets, optimize globally not independently

Generation Length Bucketing

The gen_len bucketing parameter significantly impacts padding ratio:

  1. Smaller buckets = Lower padding ratio but more batches (higher cost)
  2. Larger buckets = Fewer batches but higher padding from variance in gen_len
  3. Find the sweet spot by computing the padding budget and working backwards

Verification Strategies

Early Constraint Checks

Immediately after generating output, verify:

1. All request IDs appear exactly once
2. Number of unique shapes ≤ limit
3. Each request's prompt_len ≤ assigned shape
4. No missing shapes that would cause alignment failures

Metric Validation

Before considering a solution complete:

  1. Run the official evaluation script (if provided)
  2. Compare all metrics against all thresholds
  3. Check each bucket independently - passing one bucket does not guarantee passing others

Common Verification Failures

Watch for these issues:

  • Missing shapes that cause coverage gaps (e.g., shape 2048 missing when needed)
  • Single-request batches that waste per-batch overhead
  • Shape constraints violated when optimizing buckets independently

Common Pitfalls

Premature Optimization

  • Mistake: Jumping into implementation before understanding mathematical constraints
  • Fix: Spend time upfront computing exact budgets (e.g., "bucket_1 can tolerate at most 25,735 padding tokens")

Insufficient Cost Analysis

  • Mistake: Not understanding which cost component dominates
  • Fix: Compute and document the full cost breakdown before optimizing

Independent Bucket Optimization

  • Mistake: Optimizing each bucket separately when constrain

End of content truncated in source

Frequently Asked Questions about llm-inference-batching-scheduler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference batching to reduce cost while meeting latency constraints?▼

LLM inference batching optimization groups requests into batches while minimizing per-batch overhead, shape compilation costs, and padding waste. Analyze hard constraints first—maximum unique shapes, latency thresholds, padding limits—then decompose costs into overhead, shape compilation, prefill, and decode components. Use systematic evaluation with constraint verification to find parameter combinations that balance these trade-offs.

What's the difference between scheduling by sequence length buckets versus generation length buckets?▼

Sequence length bucketing groups requests by prompt length into shapes that minimize padding for common lengths. Generation length bucketing controls variance in output lengths within batches—smaller buckets reduce padding ratio but increase batch count and overhead; larger buckets reduce batches but increase padding from length variance. Optimize both jointly to find the cost-latency sweet spot.

How do I verify that my batching schedule meets all constraints?▼

Verify immediately after generating output: confirm each request ID appears exactly once, unique shape count stays within limits, each request's prompt length fits its assigned shape, and no coverage gaps exist. Run evaluation scripts comparing all metrics against thresholds independently per bucket. Single-bucket compliance doesn't guarantee overall feasibility.

What happens if I optimize each request bucket independently instead of globally?▼

Independent bucket optimization often violates global constraints—particularly the maximum unique shapes limit. When shapes are shared across buckets, optimizing each bucket separately can exceed the total allowed shapes or create inefficient shape distribution. Optimize all buckets jointly while tracking the global constraint budget.

How do I choose between grid search and binary search for batching parameters?▼

Use grid search for bounded, discrete parameter spaces like generation bucket sizes in range [15, 50] to systematically compare all combinations and understand trade-offs. Use binary search when optimizing a single parameter in isolation to find the optimal value efficiently. Document the optimization landscape—which combinations produce which metrics—to understand constraints.

What's a common reason batching optimizations fail to meet padding ratio limits?▼

Premature optimization before computing exact budgets causes failures. Calculate upfront how much padding each bucket can tolerate given constraints, then work backward to set generation bucket sizes. Insufficient cost analysis—not understanding which component dominates—also leads to solutions that fail on padding or latency metrics.

Related Skills

Explore More Software Engineering Skills→
bianhaifeng789-huebianhaifeng789-hue

llm-inference-batching-scheduler

Optimize LLM inference batching to minimize compilation overhead and padding waste.

Community
Advanced
saeed-vayghansaeed-vayghan
30

machine-learning-engineer

Deploy and serve machine learning models with optimized inference performance.

Community
Advanced
lazyFrogLOLlazyFrogLOL
127

llm-inference-batching-scheduler

Design LLM inference batching schedulers balancing compilation cost and latency constraints.

Community
Advanced
MicrockMicrock
322

Convex Agents Usage Tracking

Track LLM token usage and calculate costs for billing reports.

Community
Advanced