skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

huggingface-tokenizers

Train and apply fast BPE, WordPiece, and Unigram tokenizers with alignment tracking.

Updated Jun 5, 2026
Search Tags:#nlp#tokenization#huggingface#transformers#bpe#wordpiece#unigram
One-click install
npx skills add https://github.com/xu1713/openhorse --skill huggingface-tokenizers-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-tokenizers
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/huggingface-tokenizers
Command: npx skills add https://github.com/xu1713/openhorse --skill huggingface-tokenizers-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tokenizers, transformers, datasets, and includes references (resource) components.

What problem does it solve? Tokenizing large text corpora with pure Python implementations is slow, and building custom vocabularies for domain-specific or multilingual models requires deep knowledge of subword algorithms. This Skill provides guidance for using the Rust-based HuggingFace Tokenizers library to tokenize 1GB of text in under 20 seconds and train custom tokenizers from scratch. ## Core Features & Use Cases - High-Performance Tokenization: Encode text at roughly 4GB per minute using the Rust core, with batch encoding, padding, truncation, and multi-processing support. - Custom Tokenizer Training: Train BPE, WordPiece, or Unigram tokenizers on files, iterators, or streaming datasets with configurable vocabulary size, special tokens, and normalization pipelines. - Alignment Tracking & Transformers Integration: Map tokens back to original character offsets for NER and QA tasks, and wrap custom tokenizers with PreTrainedTokenizerFast for use with any transformers model. - Use Case: Train a 30k-vocabulary BPE tokenizer on a domain corpus, add BERT-style [CLS]/[SEP] post-processing, then load it via AutoTokenizer to feed a downstream classification model. ## Quick Start Ask the AI to train a BPE tokenizer with a 30,000-token vocabulary on your corpus file and show how to encode a sample sentence with padding enabled.

Frequently Asked Questions about huggingface-tokenizers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a custom BPE tokenizer in Python?▼

Initialize a Tokenizer with the BPE model, set a pre-tokenizer like Whitespace or ByteLevel, then call train() with a BpeTrainer configured with vocab_size and special tokens. Training on 100MB of text takes roughly 1-2 minutes on a multi-core CPU.

What is the difference between BPE, WordPiece, and Unigram tokenization?▼

BPE merges the most frequent character pairs and is used by GPT-2 and RoBERTa. WordPiece scores merges by likelihood ratio and powers BERT. Unigram starts with a large vocabulary and prunes tokens probabilistically, suiting multilingual models like T5.

How do I use a custom tokenizer with transformers AutoTokenizer?▼

Save the trained tokenizer to JSON, then wrap it with PreTrainedTokenizerFast specifying special tokens like unk_token and pad_token. After save_pretrained(), it loads with AutoTokenizer.from_pretrained like any Hub tokenizer.

Does HuggingFace Tokenizers support tracking token positions in original text?▼

Yes, fast tokenizers return offset mappings that link each token to character positions in the source text. This supports tasks like named entity recognition and question answering where predictions must map back to original spans.

Why is my tokenizer producing too many unknown tokens?▼

High unknown rates usually mean the vocabulary is too small or min_frequency is too high during training. Increase vocab_size, lower min_frequency, or switch to byte-level BPE which eliminates unknown tokens entirely.

When should I use SentencePiece or tiktoken instead of HuggingFace Tokenizers?▼

Use SentencePiece for language-independent tokenization required by T5 or ALBERT checkpoints, and tiktoken when matching OpenAI GPT model tokenization exactly. HuggingFace Tokenizers is preferable for training custom vocabularies and transformers integration.

Related Skills

Explore More Software Engineering Skills→
PlaidmustachePlaidmustache

nemo-curator

GPU-accelerate multimodal data curation for LLM training with RAPIDS.

Community
Intermediate
jinzhezenggroupjinzhezenggroup
124

deepmd-finetune-dpa3

Fine-tunes a pre-trained DPA3 model on a downstream dataset using DeePMD-kit.

Official
Advanced
rocklambrosrocklambros
2

auditing-rlhf-reward-hacking

Audit RLHF, DPO, and RLAIF checkpoints for reward hacking by comparing reward win-rates with preference win-rates.

Community
Advanced
box755box755

sentencepiece

Train SentencePiece BPE or Unigram tokenizers on raw Unicode text.

Community
Intermediate