count-dataset-tokens

Token counting across datasets involves loading your data, selecting the appropriate tokenizer, and aggregating token counts across all text fields. This Skill automates that process—it discovers text fields in your dataset, applies your chosen tokenizer to each entry, handles batching with progress checkpoints, and delivers total token counts with verification.

Yes. This Skill supports domain and category filtering, allowing you to count tokens only within specific subsets of your dataset rather than across the entire corpus. This is useful when you need token estimates for particular content types or splits.

You specify the tokenizer explicitly—the Skill applies it to your dataset's text fields. This gives you control over which tokenization scheme matches your model or use case, whether that's a standard HuggingFace tokenizer or a domain-specific variant.

The Skill discovers your dataset's schema automatically and tokenizes all relevant text fields per entry. It then aggregates counts across fields and entries, so you get a complete token tally for mixed-format datasets without manual field selection.

The Skill includes batch processing with progress checkpoints and edge-case handling. It documents assumptions throughout the counting process and surfaces any anomalies, so you can verify results and resume from checkpoints if needed rather than restarting from scratch.

Token counts vary significantly based on the tokenizer used—different tokenizers produce different counts for the same text. Using the exact tokenizer your model will use ensures accurate token budgeting and prevents surprises during processing or training.