knowledge-distillation

Knowledge distillation compresses large language models by training a smaller student model to imitate a larger teacher, enabling deployment with reduced compute while preserving performance.

Reverse KL divergence, applied in MiniLLM variants during teacher-student distillation, improves generation diversity and robustness for open-ended tasks by altering the training objective.

To distill a large language model, load a teacher model, initialize a smaller student, and train using temperature-scaled soft targets to transfer the teacher's knowledge.

Yes, knowledge distillation applies to model compression, domain adaptation, and fast deployment scenarios across generation tasks, multi-teacher setups, and specialized domains.

Poor performance on open-ended generation tasks during distillation often relates to using standard KL divergence; applying reverse KL or MiniLLM variants can improve generation diversity and robustness.

Evaluating a distilled deployable language model requires careful consideration of data selection, hyperparameters, training loops, and evaluation metrics to ensure reliable performance.