token_num_filter

Filter text samples by token count using a HuggingFace tokenizer to retain only samples within a specified minimum and maximum token range. This process removes overly short or long text entries to streamline dataset preprocessing.

The best way to remove overly short or long text entries is token range filtering, which calculates token counts via a HuggingFace tokenizer and retains only samples matching your defined limits for cleaner dataset trimming.

Yes, you need both py-data-juicer and transformers to filter text by token range. These dependencies provide the necessary tokenizer and filtering framework to calculate token counts and apply range limits to text samples.

Yes, you can set a minimum token limit without a maximum limit for text filtering. The filter retains samples with token counts above the specified minimum, allowing unlimited text length for samples exceeding the threshold.

Token count filtering works with JSONL files for text preprocessing by reading text samples, calculating token counts using a HuggingFace tokenizer, and outputting only the samples that fall within the specified token range.

Use token counting instead of character length when filtering text samples for model training, as tokenization aligns with how models process language and ensures samples match exact model input constraints rather than arbitrary character limits.