transformers-config-tokenizers-expert

Detect tokenizer config drift by loading a HuggingFace Transformers snapshot and running an audit across tokenizer_config.json, generation_config.json, config.json, tokenizer.json, and chat_template.jinja to surface EOS token drift, added_tokens shapes, and chat_template inconsistencies.

Check chat_template drift by validating the chat_template.jinja sidecar file against the inline chat_template defined in tokenizer_config.json. This surfaces loading behavior issues and template rendering inconsistencies across different model families like Qwen3, Gemma-4, and GLM-5.1.

Yes, you can audit tokenizer config drift offline. The process validates added_tokens and drift diagnostics directly against local snapshot files like tokenizer.json and generation_config.json without requiring external network access or downloading dependencies.

Incorrect generation often stems from EOS token drift or extra_special_tokens shape issues across tokenizer_config.json and generation_config.json. Diagnosing these configuration mismatches helps identify why the model terminates generation prematurely or produces malformed outputs.

The best way to validate added_tokens shapes is cross-referencing the added_tokens definitions in tokenizer.json against the extra_special_tokens in tokenizer_config.json. This surfaces shape inconsistencies across model families like Kimi-K2, Qwen3, and Gemma-4.