layer-inference-planner

Plan layer-by-layer inference by generating a structured plan using the model_id, total_layers, activation_size_mb, and available_vram_mb to distribute 70B+ models across constrained hardware. You can configure single-device sequential planning or multi-device pipeline distribution.

Estimate latency for distributed inference by applying multi-device pipeline distribution with activation transfer and latency budgeting. The planner outputs structured timing estimates and visualizations to help you compare different distribution strategies.

AirLLM-style layer-by-layer inference works with limited VRAM by planning sequential execution across mesh devices. By providing available_vram_mb and activation_size_mb, the planner structures the pipeline to run large models on constrained hardware.

The best way to run 70B models on hardware with VRAM constraints is generating a multi-device distributed plan with activation transfer. This approach pipelines layers across mesh devices to fit the model within available memory limits.

Start single-device sequential inference planning by setting the action to plan_single and providing the model_id, total_layers, activation_size_mb, and available_vram_mb parameters to generate a structured execution plan.

Limitations of layer-by-layer inference planning include dependencies on the @sven/compute-mesh/layer-inference package and the need to accurately specify activation sizes and VRAM. Multi-device pipeline distribution also requires accounting for activation transfer overhead between mesh devices.