model-infer-prefetch

First confirm memory-bound hotspots via profiling, then select target operators and non-memory-bound dependency windows, compute max_size from weight dimensions and dtype, insert torch_npu.npu_prefetch calls behind an enable_prefetch switch, and validate with before/after profiling.

Memory-bound weight-heavy operators like MatMul, BatchMatMul, QBMM, and GMM benefit most. Compute-bound operators gain nothing, and small operators like LayerNorm, ROPE, or SwiGLU serve as dependency windows rather than prefetch targets.

Pick a preceding operator on the same logical path that is not memory-bound and has enough execution time to cover the weight transfer. Large MatMul outputs can serve as windows since bandwidth frees up after they complete; avoid communication or heavy data-movement operators.

Compute max_size as weight dimension product times dtype bytes (2 for bf16, 1 for int8), then start at 50% of the theoretical value. Increase toward 70-100% if the target speeds up, or decrease if the dependency window degrades from bandwidth contention.

Graph mode requires the dependency tensor to be on the same logical path as the target operator; cross-branch dependencies cause compile failures or merely shift waiting. Oversized max_size also steals bandwidth from the dependency window, so reduce it or reposition the prefetch.

Skip prefetching when profiling shows no memory-bound bottleneck, when all candidate dependency windows are communication or heavy data-movement operators, or when the preceding path offers no safe bandwidth window. Forcing prefetch in these cases adds contention without benefit.