graph-mode-adaptation

You can accelerate model inference on NPU by adapting models to graph mode using torch.compile. This requires an NPU-enabled environment with torch_npu installed, addressing graph breaks, dynamic shapes, and backend-specific constraints to ensure both stability and performance.

The npugraph_ex and GE graph modes are two distinct backends for NPU acceleration. Choosing the right backend depends on your specific model requirements, as each handles graph compilation and execution differently, requiring a structured workflow for design and validation.

To fix graph breaks and recompilation issues during NPU graph mode adaptation, you must identify and resolve unsupported operations and dynamic shape changes. A structured debugging workflow helps isolate these backend-specific constraints to ensure stable model execution.

Yes, you must separate prefill and decode paths when adapting LLMs for NPU graph mode. This separation is a key requirement for LLM models to properly handle dynamic shapes and static graph considerations during torch.compile acceleration.

Using torch.compile with NPU for dynamic shapes introduces limitations such as frequent recompilation and graph breaks. You must apply static graph considerations and backend-specific constraints to manage these issues and maintain inference performance.