knowledge-distillation

Transfer knowledge from large teacher models to smaller student models.

Updated Mar 16, 2026

Search Tags:#llm #distillation #teacher-student #knowledge-distillation #minillm #soft-targets #temperature-scaling

Ready to run

npx skills add https://github.com/arsity/scholar-tools --skill knowledge-distillation-arsity

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: knowledge-distillation
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/19-emerging-techniques/knowledge-distillation
Command: npx skills add https://github.com/arsity/scholar-tools --skill knowledge-distillation-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, datasets, and includes references (resource) components.

What problem does it solve?

Knowledge distillation enables deploying smaller, efficient language models by transferring capabilities from large teachers to compact students while preserving performance.

Core Features & Use Cases

Techniques: temperature scaling, soft targets, reverse KL (MiniLLM), and logit distillation to guide student learning.
Strategies: single-teacher, two-stage, and multi-teacher distillation for robust performance across tasks.
Use Case: compress a 70B model to 7B while maintaining accuracy for production inference.

Quick Start

Install the required libraries and run the distillation pipeline to train a smaller student model from a larger teacher.

Frequently Asked Questions about knowledge-distillation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I compress a large LLM to a smaller model using teacher-student distillation?▼

Teacher-student distillation compresses large language models by transferring capabilities to compact students using temperature scaling and soft targets. This Skill executes end-to-end logit distillation pipelines to preserve performance while reducing model footprint for production inference.

What is reverse KL distillation and when should I use it for LLM compression?▼

Reverse KL distillation, implemented here as MiniLLM, is a technique guiding student model learning by optimizing logit distributions. Use it when standard forward KL causes the student to over-generate high-probability tokens from the teacher, ensuring more robust performance across NLP tasks.

Does this knowledge distillation pipeline work with Hugging Face transformers and PyTorch?▼

Yes, this pipeline requires the transformers, torch, and datasets libraries for training and evaluation. It integrates directly with these frameworks to process datasets and manage the teacher-student model training loop end-to-end.

Can I use multi-teacher distillation to compress a 70B model down to 7B?▼

Yes, multi-teacher distillation is supported to compress large models like a 70B parameter teacher down to a 7B student. This strategy leverages multiple teacher models to robustly train a smaller student while maintaining accuracy for production inference.

What's the best way to apply temperature scaling and soft targets for model compression?▼

Temperature scaling and soft targets are applied during the logit distillation process to smooth teacher output distributions before transferring knowledge. This Skill handles this automatically, using these techniques within single-teacher or two-stage distillation strategies to guide student learning.

Why does my student model lose accuracy compared to the teacher during knowledge distillation?▼

Student accuracy loss during knowledge distillation often stems from insufficient temperature scaling or relying solely on hard labels. Applying reverse KL (MiniLLM) and multi-teacher strategies with soft targets helps minimize this performance gap while reducing footprint.