sentencepiece

Tokenize raw Unicode text by training a SentencePiece model using BPE or Unigram algorithms with configurable vocab_size and character_coverage. This generates a .model file to encode and decode text into deterministic or sampled subword pieces for transformer pipelines.

Subword regularization is a data augmentation technique that samples multiple subword segmentations probabilistically. By enabling sampling options during encoding, it generates varied subword pieces from the same text to improve model robustness during training.

Yes, it supports language-independent tokenization for CJK languages without pre-tokenization. It treats whitespace as a regular symbol, enabling consistent subword segmentation across multilingual corpora and scripts without language-specific preprocessing.

Choose BPE for deterministic subword segmentation based on merge operations, or Unigram for probabilistic segmentation that enables subword regularization sampling. Both algorithms support configurable vocab_size and character_coverage for multilingual corpus training.

Yes, it supports training T5-style or mBART tokenizers. You configure the model_type, vocab_size, and character_coverage on your raw text corpus to produce a reproducible .model file for consistent preprocessing in transformer pipelines.