knowledge-distillation

What problem does it solve?

This Skill addresses the high inference costs and memory requirements associated with deploying large, powerful LLMs. It enables you to compress these models into smaller, more efficient versions while retaining a significant portion of their original performance.

Core Features & Use Cases

• Model Compression: Reduce model size (e.g., from 70B to 7B parameters) while retaining 90%+ of the larger teacher model's performance.
• Capability Transfer: Distill advanced capabilities and nuanced knowledge from proprietary models (like GPT-4) into smaller, open-source student models.
• Inference Cost Reduction: Significantly lower operational costs by deploying efficient student models that require less compute and memory.
• Specialized Models: Create smaller, domain-specific models by distilling targeted knowledge from a general-purpose teacher.
• Use Case: Transform a powerful but expensive 70B parameter model into a 7B parameter model that can run on a single GPU, drastically reducing API costs for a high-volume application.

Quick Start

Distill knowledge from a Llama-2-70b-hf teacher to a Llama-2-7b-hf student using a combined loss of 70% soft (KL divergence) and 30% hard (cross-entropy) with a temperature of 2.0.