sentencepiece

Train and apply language-independent BPE or Unigram tokenizers on raw Unicode text.

13.0k|930|Updated Nov 3, 2025

Search Tags:#nlp #multilingual #tokenization #sentencepiece #bpe #unigram #subword

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill sentencepiece-orchestra-research

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sentencepiece
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/02-tokenization/sentencepiece
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill sentencepiece-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sentencepiece, transformers, and includes references (resource) components.

What problem does it solve? Tokenizing multilingual or CJK text usually requires language-specific pre-tokenization rules that are brittle and hard to reproduce. SentencePiece trains tokenizers directly on raw text, producing deterministic vocabularies that work across all languages. ## Core Features & Use Cases - BPE and Unigram Training: Train subword tokenizers on raw corpora with configurable vocabulary size, character coverage, and special tokens. - Language-Independent Encoding: Treat text as raw Unicode with whitespace as a meta symbol, supporting Chinese, Japanese, Korean, and multilingual corpora without preprocessing. - Subword Regularization: Sample alternative tokenizations during training for data augmentation and model robustness. - Use Case: Train a 32k Unigram tokenizer on a multilingual corpus for a T5-style model, then integrate it with HuggingFace Transformers via T5Tokenizer. ## Quick Start Train a SentencePiece Unigram tokenizer with 8000 vocabulary on my corpus.txt file and show me how to encode and decode sample sentences with it.

Frequently Asked Questions about sentencepiece

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I train a SentencePiece tokenizer in Python?▼

Use spm.SentencePieceTrainer.train with your input corpus file, model prefix, vocab_size, and model_type set to 'unigram' or 'bpe'. Training a 100MB corpus takes roughly 1-4 minutes depending on the algorithm.

SentencePiece BPE vs Unigram: which should I use?▼

Unigram is probabilistic, supports sampling for subword regularization, and suits multilingual tasks; it is used by T5, ALBERT, and XLNet. BPE trains faster and is deterministic, used by mBART.

Does SentencePiece work with Chinese, Japanese, and Korean text?▼

Yes, SentencePiece is language-independent and handles CJK text without pre-tokenization. Set character_coverage=1.0 when training on CJK corpora to ensure all characters are covered.

Can I use SentencePiece with HuggingFace Transformers?▼

Yes, tokenizers like T5Tokenizer load SentencePiece models internally. You can load pretrained models such as t5-base via T5Tokenizer.from_pretrained and encode text directly.

When should I not use SentencePiece?▼

For faster training and more flexibility, HuggingFace Tokenizers tokenizes about 4x faster. For OpenAI models use tiktoken, and for English-centric BERT tasks WordPiece may be preferable.