Skills
.Work. You
Rest

token_num_filter

Official

Filter text samples based on token count within specified range

Data & Analytics#token counting#text filtering#sample selection#text preprocessing#token range
Authorcas-bigdatalab
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill solves the problem of filtering text samples based on their token count, allowing users to retain samples within a specified range of token numbers.

Core Features & Use Cases

  • Token Count Filtering: Utilizes HuggingFace tokenizer to calculate token counts of text samples.
  • Range-Based Filtering: Retain samples with token counts within the specified minimum and maximum limits.
  • Use Case: When working with large text datasets and needing to filter out samples that are too short or too long.

Quick Start

Apply the token_num_filter skill to the 'example_input.jsonl' file, keeping text samples with a minimum of 10 tokens and no maximum limit.

Dependency Matrix

Required Modules

py-data-juicertransformers

Components

scripts

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: token_num_filter
Download link: https://github.com/cas-bigdatalab/piflow/archive/main.zip#token-num-filter

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.