nanogpt

Prepare a dataset with the provided prepare.py script to create train.bin and val.bin token files, then run train.py with a config file. For a quick start, train a character-level model on Shakespeare in about five minutes on CPU.

Set init_from to gpt2, gpt2-medium, gpt2-large, or gpt2-xl in your config to load OpenAI weights automatically. Use a lower learning rate around 3e-5, add dropout for regularization, and point the dataset parameter at your prepared token files.

nanoGPT is best for learning and experimentation since the entire model and training loop fit in about 600 lines of hackable PyTorch. HuggingFace Transformers suits production use with many pretrained architectures and high-level APIs.

Yes, the character-level Shakespeare config trains on CPU in about five minutes with a small 6-layer, 384-dimensional model. Larger models like GPT-2 124M require GPUs, roughly four days on eight A100s.

CUDA out of memory occurs when batch size or context length exceeds VRAM. Reduce batch_size and block_size, increase gradient_accumulation_steps to keep the effective batch, and enable bfloat16 dtype to cut memory usage roughly in half.

nanoGPT lacks the abstractions, model zoo, and deployment tooling of production frameworks. For large-scale distributed training use Megatron-LM, and for production-ready multi-architecture support consider LitGPT or HuggingFace Transformers.