skills.rest
debug-zhuweijian
debug-zhuweijianCOMMUNITY
Explore all skills by this author →
GitHub

knowledge-distillation

Compress a large teacher model into a smaller student via knowledge distillation.

13|1|Updated Apr 14, 2026
Search Tags:#knowledge distillation#temperature scaling#model compression#soft targets#teacher-student#minillm#reverse kl
One-click install
npx skills add https://github.com/debug-zhuweijian/ai-research-toolkit --skill knowledge-distillation-debug-zhuweijian
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: knowledge-distillation
Source: https://github.com/debug-zhuweijian/ai-research-toolkit/tree/main/modules/05-knowledge/skills/knowledge-distillation
Command: npx skills add https://github.com/debug-zhuweijian/ai-research-toolkit --skill knowledge-distillation-debug-zhuweijian

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, datasets, and includes references (resource) components.

What problem does it solve?

Knowledge distillation reduces the size of language models by transferring knowledge from a large teacher model to a smaller student, enabling faster inference, lower resource usage, and easier deployment without substantial loss in performance.

Core Features & Use Cases

  • Two-stage distillation workflows combining teacher-guided training and task-specific fine-tuning.
  • Support for forward KL and reverse KL (MiniLLM) strategies, temperature scaling, and soft targets.
  • Multi-teacher distillation and logits-level distillation options.
  • Practical deployment guidance for compressing 70B to 7B or similar scales.

Quick Start

Run a basic knowledge distillation workflow to compress a large teacher model into a smaller student while preserving performance.

Frequently Asked Questions about knowledge-distillation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compress a large language model for faster inference without losing performance?▼

Knowledge distillation compresses a large teacher model into a smaller student, transferring learned behaviors to enable faster inference and easier deployment without substantial performance loss.

When should I use reverse KL divergence instead of forward KL for model distillation?▼

Reverse KL divergence, supported via the MiniLLM strategy, helps generate more focused output distributions by avoiding mode-covering behavior, whereas forward KL covers broader teacher distributions during knowledge distillation.

Can I distill a 70B parameter teacher model down to a 7B student model?▼

Yes, this skill provides practical deployment guidance and hyperparameter configuration for large-scale compression tasks, including distilling a 70B teacher model into a 7B student model.

How do I configure temperature scaling and alpha for soft target distillation?▼

You must configure distillation hyperparameters including temperature and alpha to effectively soften target logits, balancing the learning signal between the teacher model's soft targets and ground truth labels.

Does knowledge distillation work with multi-teacher ensembles and logits-level training?▼

Yes, the workflow supports multi-teacher distillation and logits-level distillation options, allowing you to aggregate knowledge from several teacher models to train a single student model.

What is a two-stage distillation workflow for large language models?▼

Two-stage distillation combines teacher-guided training on soft targets with subsequent task-specific fine-tuning, ensuring the student model learns general representations before adapting to specific downstream tasks.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

llama-factory

Automate LLaMA-family model fine-tuning via the Llama-Factory WebUI.

Community
Advanced
Supporter09Supporter09

knowledge-distillation

Compress large language models via teacher-to-student knowledge distillation.

Community
Advanced
arsityarsity

knowledge-distillation

Transfer knowledge from large teacher models to smaller student models.

Community
Advanced
tianhao909tianhao909
1

model-pruning

Prune LLMs with Wanda and SparseGPT to reduce size and accelerate inference.

Community
Advanced