skills.rest
zhang-ming-hui
zhang-ming-huiCOMMUNITY
Explore all skills by this author →
GitHub

dataset-cleaning

Clean, normalize, deduplicate, and validate scraped skill records.

Updated May 28, 2026
Search Tags:#indexing#information retrieval#data validation#normalization#deduplication#dataset cleaning#scraped data
One-click install
npx skills add https://github.com/zhang-ming-hui/ackownledge --skill dataset-cleaning
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataset-cleaning
Source: https://github.com/zhang-ming-hui/ackownledge/tree/main/ir_system/.agents/skills/dataset-cleaning
Command: npx skills add https://github.com/zhang-ming-hui/ackownledge --skill dataset-cleaning

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Scraped skill records often contain duplicates, malformed fields, and inconsistent normalized values that degrade indexing quality and retrieval reliability.

Core Features & Use Cases

  • Deterministic record normalization: Applies repeatable rules to standardize formats while preserving the underlying meaning.
  • Deduplication and validation for indexing compatibility: Reduces duplicate/malformed entries and ensures the cleaned dataset still matches the index-ready record shape.
  • Checkpoint-friendly repair workflow: Uses crawl checkpoints and sample datasets to safely improve quality without losing traceability.

Quick Start

Use the dataset-cleaning skill to clean skills_data_500.json/csv using skills_checkpoint.json and produce safer, index-compatible cleaned output with consistent normalized fields.

Frequently Asked Questions about dataset-cleaning

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean scraped data to fix duplicates and malformed fields before indexing?▼

To clean scraped data for indexing, you apply deterministic normalization rules to standardize formats, deduplicate records, and validate fields to ensure the output dataset is index-compatible and retrieval-ready.

What is the best way to normalize scraped skill records without losing stable URLs and identifiers?▼

The best way to normalize scraped skill records is using deterministic cleaning logic that standardizes field formats while explicitly preserving stable identifiers and URLs to maintain data traceability and retrieval accuracy.

How does data validation improve information retrieval quality for normalized datasets?▼

Data validation improves information retrieval quality by removing malformed entries and ensuring normalized values match the index-ready record shape, which directly increases search accuracy and evaluation reliability.

Can I use crawl checkpoints to repair scraped data without losing traceability?▼

Yes, you can use crawl checkpoints and sample datasets to execute a checkpoint-friendly repair workflow, allowing you to safely improve dataset quality without losing traceability of the original scraped records.

Does dataset cleaning work with both JSON and CSV inputs for scraped skill records?▼

Yes, dataset cleaning processes both JSON and CSV scraped skill record formats, applying consistent normalization and deduplication rules to produce a safer, index-compatible cleaned output for your search system.

Related Skills

Explore More Data & Analytics Skills→
jackandkingjackandking

data-deduplication

Deduplicate datasets using exact, fuzzy, ID-based, and content similarity strategies.

Community
Intermediate
nicholasglazernicholasglazer
27

eval

Evaluate corpus retrieval quality with Hit@5, MRR, and nDCG metrics.

Community
Intermediate
buildinternetbuildinternet
3

finding-changelogs

Evaluate changelog URLs and recommend optimal ingestion methods.

Official
Advanced
Adelie-SquadAdelie-Squad
17

data-engineer

Clean and transform raw datasets into analysis-ready tables with pandas pipelines.

Official
Advanced