transformers-config-tokenizers-expert

What problem does it solve?

The skill helps engineers identify and reason about tokenizer/config drift across HuggingFace Transformers snapshots, surfacing risks such as EOS token drift, added_tokens shapes, chat_template loading issues, and drift between tokenizer_config.json, tokenizer.json, and generation_config.json.

Core Features & Use Cases

• Surface EOS/token drift across tokenizer_config.json, generation_config.json, and config.json.
• Detect extra_special_tokens shape issues and chat_template drift between sidecar and inline templates.
• Provide guidance for offline and air-gapped usage, including how to verify added_tokens and drift diagnostics.

Quick Start

Load a HF snapshot and run a drift audit across tokenizer_config.json, generation_config.json, config.json, tokenizer.json, and chat_template.jinja to surface drift.