huggingface-tokenizers

Fast tokenization uses a Rust core with Python bindings to accelerate large-scale NLP pipelines. This approach supports BPE, WordPiece, and Unigram models to process text efficiently for production environments.

You can wrap trained tokenizers for Transformers using PreTrainedTokenizerFast and AutoTokenizer. This interoperability allows custom BPE, WordPiece, or Unigram tokenizers to function directly within the Transformers framework.

Supported tokenization models include BPE, WordPiece, and Unigram. These models can be combined with flexible tokenization stacks featuring robust normalizers, pre-tokenizers, post-processors, and decoders for custom NLP pipelines.

Alignment tracking provides offset mapping to link tokens back to original text positions. This ensures reliable downstream task performance by maintaining character-level alignment during multilingual text processing and decoding.

You do not need Rust installed because the tokenizers package provides Python bindings over a compiled Rust core. Install the package directly to access production-ready tokenization tooling without managing Rust toolchains.

The best way to train a custom tokenizer is to install the tokenizers package, select a model like BPE or Unigram, train it on your corpus, and wrap the result with PreTrainedTokenizerFast for framework integration.