knowledge-distillation

Teacher-student distillation compresses large language models by transferring capabilities to compact students using temperature scaling and soft targets. This Skill executes end-to-end logit distillation pipelines to preserve performance while reducing model footprint for production inference.

Reverse KL distillation, implemented here as MiniLLM, is a technique guiding student model learning by optimizing logit distributions. Use it when standard forward KL causes the student to over-generate high-probability tokens from the teacher, ensuring more robust performance across NLP tasks.

Yes, this pipeline requires the transformers, torch, and datasets libraries for training and evaluation. It integrates directly with these frameworks to process datasets and manage the teacher-student model training loop end-to-end.

Yes, multi-teacher distillation is supported to compress large models like a 70B parameter teacher down to a 7B student. This strategy leverages multiple teacher models to robustly train a smaller student while maintaining accuracy for production inference.

Temperature scaling and soft targets are applied during the logit distillation process to smooth teacher output distributions before transferring knowledge. This Skill handles this automatically, using these techniques within single-teacher or two-stage distillation strategies to guide student learning.

Student accuracy loss during knowledge distillation often stems from insufficient temperature scaling or relying solely on hard labels. Applying reverse KL (MiniLLM) and multi-teacher strategies with soft targets helps minimize this performance gap while reducing footprint.