model-pruning

Pruning an LLM without retraining is possible using one-shot methods like Wanda and SparseGPT. These techniques achieve high sparsity, such as 50%, with minimal accuracy degradation, reducing model size and computational cost for efficient deployment on resource-constrained hardware.

Wanda and SparseGPT are both one-shot pruning methods for LLMs. Wanda prunes weights based on magnitude and input activations, while SparseGPT uses a sparse regression approach to compress models. Both achieve high sparsity without retraining, but differ in their internal pruning mechanisms.

Yes, you can use N:M sparsity patterns to accelerate inference on hardware accelerators. These hardware-friendly sparsity patterns enable 2-4x inference speedups by structuring the sparsity to match accelerator capabilities, making them ideal for real-time applications like chatbots.

When pruning a Llama-2 7B model to 50% sparsity using Wanda, you should expect minimal accuracy degradation. The method is designed to preserve model performance while cutting size in half, enabling faster response times in applications like real-time chatbots.

Yes, model pruning supports both structured and unstructured sparsity, alongside N:M and magnitude pruning. This flexibility enables LLM deployment on mobile or edge devices with limited memory and processing power by tailoring the sparsity pattern to specific hardware constraints.

To run LLM pruning scripts, you need PyTorch, transformers, accelerate, datasets, and lm_eval. These dependencies provide the foundational model loading, acceleration utilities, evaluation datasets, and benchmarking frameworks required to execute and assess the pruning workflow.