model-pruning

Prune LLM weights using Wanda and SparseGPT to reduce model size and accelerate inference.

13.0k|930|Updated Nov 3, 2025

Search Tags:#pytorch #sparsity #inference optimization #llm compression #model pruning #wanda #sparsegpt

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill model-pruning-orchestra-research

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: model-pruning
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/19-emerging-techniques/model-pruning
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill model-pruning-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, and includes references (resource) components.

What problem does it solve? Large language models are expensive to serve and too big for constrained hardware. This Skill compresses LLMs by removing redundant weights, achieving 40-60% size reduction with minimal accuracy loss and enabling faster inference on hardware accelerators. ## Core Features & Use Cases - One-Shot Pruning: Apply Wanda (weights × activations) or SparseGPT (second-order) to prune models to 50% sparsity without any retraining. - Hardware-Friendly Sparsity: Implement N:M structured pruning (2:4, 4:8) compatible with NVIDIA sparse tensor cores for 2× inference speedup. - Production Pipelines: Gradual, layer-wise, and iterative pruning strategies with fine-tuning and lm-eval based evaluation. - Use Case: Compress LLaMA-2-7B to 50% sparsity using Wanda with 128 calibration samples, then evaluate on ARC and HellaSwag to confirm under 1% accuracy degradation before deployment. ## Quick Start Prune my LLaMA-2-7B model to 50% sparsity using Wanda and evaluate the accuracy loss on standard benchmarks.

Frequently Asked Questions about model-pruning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I prune a LLaMA model without retraining?▼

Use Wanda or SparseGPT one-shot pruning with a small calibration dataset of about 128 samples. Wanda computes importance as weight magnitude times input activation norm, then removes the lowest-importance weights to reach your target sparsity in minutes.

Wanda vs SparseGPT: which pruning method should I use?▼

Wanda is simpler and faster with O(n) complexity per layer, losing about 0.8% accuracy at 50% sparsity. SparseGPT uses second-order Hessian information for slightly better quality (0.4% loss) but is slower and more memory-intensive. Use Wanda unless you need the absolute best quality.

Does pruning actually speed up LLM inference?▼

Unstructured pruning reduces memory by 50% but gives no speedup due to irregular sparsity. N:M structured pruning (2:4 or 4:8) achieves 2× speedup on NVIDIA GPUs with sparse tensor cores, such as the A100.

What sparsity level is safe for LLM pruning?▼

30% sparsity is conservative with under 0.5% accuracy loss, while 50% is the recommended balance at roughly 1% loss. Beyond 70% sparsity, expect 2-5% degradation, and 90% causes significant quality loss unless done iteratively with fine-tuning.

Why does one-shot pruning fail at high sparsity?▼

Removing too many weights at once destroys model capacity without recovery. Use gradual magnitude pruning or iterative prune-and-finetune cycles, increasing sparsity in steps with short fine-tuning runs between iterations to recover accuracy.