nanogpt

NanoGPT provides a minimal, educational GPT implementation in roughly 300 lines of PyTorch. This compact codebase allows for easy inspection and modification of the model definition and training loop without the complexity of large frameworks.

You can train a minimal GPT by preparing data with character-level or BPE tokenization and running the provided small-model training configuration. This pipeline offers an end-to-end recipe for hands-on learning and quick prototyping on small datasets like Shakespeare.

Yes, you can reproduce GPT-2 (124M) scale experiments on multi-GPU setups. The implementation supports distributed data parallel (DDP), mixed-precision (bf16/fp16), and torch compilation for large-scale reproduction and fine-tuning on OpenWebText.

Yes, the GPT training pipeline supports BPE tokenization via tiktoken and mixed-precision training using bf16 or fp16. These features are configurable within the provided training recipes for both small experiments and GPT-2 scale reproduction.

The best way to fine-tune pretrained GPT-2 weights is using a compact PyTorch training pipeline that supports checkpointing and configurable training recipes. This allows you to load existing weights and adapt them on custom datasets while leveraging multi-GPU support.