gpt2-codegolf

GPT-2 inference in constrained environments requires implementing three core components: parsing binary checkpoints to extract weights, building a BPE tokenizer to convert text to token IDs, and executing minimal transformer forward passes. Start by validating checkpoint loading, then verify tokenization accuracy, and finally test numerical stability of the forward pass within your size budget.

Byte-pair encoding (BPE) tokenization maps text into token IDs by iteratively merging frequent byte pairs into subword units. GPT-2 uses BPE to balance vocabulary size and compression efficiency, allowing it to handle any Unicode text while keeping the token space manageable and enabling efficient inference in resource-constrained settings.

Yes. Checkpoint parsing extracts and validates weights from compact binary formats into usable tensors. In code-golf constraints, this means implementing minimal parsing logic that verifies weight shapes and data types, then loads them into memory—prioritizing correctness over compactness, then optimizing for size after validation passes.

Tight size constraints force trade-offs: you must omit advanced features like attention caching, quantization, or multi-layer parallelism. Correctness verification becomes harder to demonstrate within limited lines. Use component-wise validation—test checkpoint loading, tokenization, and forward-pass numerics independently—before optimizing for size.

Validate your BPE tokenizer against known token sequences from standard GPT-2 checkpoints. Test edge cases: special tokens, Unicode boundaries, and repeated subwords. Component-wise validation ensures your token IDs align with the model's expectations before running inference, catching tokenization bugs early.

No external dependencies are required. GPT-2 inference can be built from first principles—checkpoint parsing uses only binary I/O, BPE tokenization uses only string manipulation and frequency counting, and forward passes use only basic matrix operations—making it viable for code-golf and verification-driven workflows.