skills.rest
zechenzhangAGI
zechenzhangAGICOMMUNITY
Explore all skills by this author →
Website𝕏@ZechenZhang5GitHub

knowledge-distillation

Compress large language models by transferring teacher knowledge to smaller student models.

11.5k|842|Updated Nov 3, 2025
Search Tags:#knowledge distillation#temperature scaling#LLM compression#soft targets#inference cost#MiniLLM#model transfer
One-click install
npx skills add https://github.com/zechenzhangAGI/AI-research-SKILLs --skill knowledge-distillation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: knowledge-distillation
Source: https://github.com/zechenzhangAGI/AI-research-SKILLs/tree/main/19-emerging-techniques/knowledge-distillation
Command: npx skills add https://github.com/zechenzhangAGI/AI-research-SKILLs --skill knowledge-distillation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, datasets, accelerate, torch, deepspeed, wandb, and includes references (resource) components.

What problem does it solve?

This Skill addresses the high inference costs and memory requirements associated with deploying large, powerful LLMs. It enables you to compress these models into smaller, more efficient versions while retaining a significant portion of their original performance.

Core Features & Use Cases

  • Model Compression: Reduce model size (e.g., from 70B to 7B parameters) while retaining 90%+ of the larger teacher model's performance.
  • Capability Transfer: Distill advanced capabilities and nuanced knowledge from proprietary models (like GPT-4) into smaller, open-source student models.
  • Inference Cost Reduction: Significantly lower operational costs by deploying efficient student models that require less compute and memory.
  • Specialized Models: Create smaller, domain-specific models by distilling targeted knowledge from a general-purpose teacher.
  • Use Case: Transform a powerful but expensive 70B parameter model into a 7B parameter model that can run on a single GPU, drastically reducing API costs for a high-volume application.

Quick Start

Distill knowledge from a Llama-2-70b-hf teacher to a Llama-2-7b-hf student using a combined loss of 70% soft (KL divergence) and 30% hard (cross-entropy) with a temperature of 2.0.

Frequently Asked Questions about knowledge-distillation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compress a large language model to reduce inference costs?▼

Knowledge distillation compresses large models by transferring knowledge from a teacher model to a smaller student model using temperature scaling and soft targets. This reduces model size (e.g., 70B to 7B parameters) while retaining 90%+ performance, significantly lowering deployment costs and memory requirements.

Can I distill capabilities from GPT-4 into smaller open-source models like LLaMA?▼

Yes, knowledge distillation enables capability transfer from proprietary models like GPT-4 to smaller open-source architectures such as LLaMA and Mistral. Using logit distillation and response distillation with compatible frameworks like Transformers and PyTorch, you retain advanced reasoning in compact models.

What's the difference between soft targets and hard targets in model distillation?▼

Soft targets use temperature-scaled probability distributions (KL divergence loss) to transfer nuanced knowledge, while hard targets use standard cross-entropy loss on true labels. A combined approach—typically 70% soft and 30% hard with temperature 2.0—balances knowledge transfer with task-specific learning.

Do I need GPUs and specific frameworks like PyTorch and Transformers to run knowledge distillation?▼

Yes, knowledge distillation requires PyTorch, Transformers, and typically GPU acceleration via tools like DeepSpeed and Accelerate. You also need compatible teacher and student models, data pipelines, evaluation tooling, and monitoring via Weights & Biases for tracking training at scale.

What are the limitations of knowledge distillation for specialized or domain-specific models?▼

Distillation quality depends on teacher model capability and data relevance. Creating domain-specific models requires carefully curated or synthetic-data-enhanced distillation. Performance gains diminish with extreme size reduction, and distillation cannot exceed the teacher's original capabilities.

How does reverse KLD and MiniLLM approach differ from standard logit distillation?▼

Reverse KLD (MiniLLM) reverses the divergence direction to penalize the student for assigning low probability to teacher-preferred outputs, improving stability. This complements standard logit distillation and enables more effective transfer of complex reasoning patterns.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

distillation

Automate knowledge distillation from teacher to student models via the Tinker API.

Official
Advanced
ovachieverovachiever
52

llama-factory

Automate LLaMA-family model fine-tuning via the Llama-Factory WebUI.

Community
Advanced
RheasilviaRheasilvia

weights-and-biases

Log machine learning experiment metrics and hyperparameters to Weights & Biases.

Community
Advanced
pluginagentmarketplacepluginagentmarketplace
4

model-optimization

Optimize machine learning models with quantization, pruning, and ONNX export.

Community
Advanced