model-pruning

Prune large language models to reduce size and accelerate inference using one-shot techniques like Wanda, SparseGPT, or N:M sparsity. This shrinks the model for constrained hardware without requiring substantial retraining.

Unstructured pruning removes individual weights based on activation-aware scoring, while N:M sparsity enforces a structured pattern of zeroed weights. Both balance accuracy and speed, but N:M is often better supported by hardware accelerators.

Yes, calibration data is required for activation-aware scoring during the pruning pipeline. This data allows techniques like Wanda and SparseGPT to evaluate weight importance accurately before exporting the final pruned model.

Yes, the pruning pipeline relies on PyTorch via the torch dependency and Hugging Face via the transformers dependency. These frameworks support applying magnitude pruning and exporting the resulting sparse model.

The best way to deploy on edge devices without retraining is applying one-shot pruning to achieve target sparsity. Methods like SparseGPT and Wanda compress the model while maintaining acceptable accuracy for latency-constrained services.

Avoid one-shot pruning if your target deployment cannot handle unstructured sparsity efficiently or if your accuracy requirements are too strict to tolerate the degradation from aggressive sparsity levels without subsequent retraining.