skills.rest
thinking-machines-lab
thinking-machines-labOFFICIAL
Explore all skills by this author →
WebsiteGitHub

distillation

Automate knowledge distillation from teacher to student models via the Tinker API.

4.0k|507|Updated Jul 14, 2025
Search Tags:#distillation#llm-training#tinker-api#teacher-student#knowledge-distillation#on-policy#multi-teacher
One-click install
npx skills add https://github.com/thinking-machines-lab/tinker-cookbook --skill distillation-thinking-machines-lab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distillation
Source: https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/.claude/skills/distillation
Command: npx skills add https://github.com/thinking-machines-lab/tinker-cookbook --skill distillation-thinking-machines-lab

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Set up and run knowledge distillation from a teacher model to a student model using the Tinker API. Use when you want to compress models or transfer knowledge from teacher to student.

Core Features & Use Cases

  • On-policy distillation: student generates data while the teacher supervises via KL penalty.
  • Off-policy distillation: SFT on reasoning traces or curated datasets.
  • Multi-teacher distillation: combine multiple teachers on diverse datasets.

Quick Start

Run the on-policy distillation recipe to train a student model from a teacher using the Tinker API.

Frequently Asked Questions about distillation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up knowledge distillation from a teacher to a student model?▼

Knowledge distillation from a teacher to a student model is set up by configuring a TeacherConfig, DistillationDatasetConfig, and dataset builders to automate training loops via the Tinker API.

What is the difference between on-policy and off-policy distillation?▼

On-policy distillation involves the student generating data while the teacher supervises via a KL penalty, whereas off-policy distillation performs supervised fine-tuning on reasoning traces or curated datasets.

Can I combine multiple teachers for knowledge distillation?▼

Multi-teacher distillation allows you to combine multiple teachers on diverse datasets, transferring and compressing knowledge into a single student model using the Tinker API.

How do I compress a large model using knowledge distillation?▼

To compress a large model, you transfer knowledge from a teacher to a smaller student model by running on-policy, off-policy, or multi-teacher distillation workflows configured through the Tinker API.

Do I need specific dataset configurations to run a distillation workflow?▼

You need a DistillationDatasetConfig and dataset builders to properly configure datasets and training loops for executing on-policy, off-policy, or multi-teacher knowledge distillation.

Related Skills

Explore More Software Engineering Skills→
FVossebeldFVossebeld

book-sft-pipeline

Convert books into SFT datasets and orchestrate LoRA training pipelines.

Community
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
sovr610sovr610

V-JEPA 2 Self-Supervised Training

Train V-JEPA 2 models with EMA target encoder and checkpointing.

Community
Advanced
jaydubya818jaydubya818

book-sft-pipeline

Create supervised fine-tuning datasets from ePub books and train LoRA style-transfer models.

Community
Advanced