skills.rest
Bhanunamikaze
BhanunamikazeCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

dataset-generator

Generate SFT and DPO training datasets from topics, URLs, and local files.

18|4|Updated Mar 24, 2026
Search Tags:#verification#grounding#deduplication#export#dpo#sft#dataset generation
One-click install
npx skills add https://github.com/Bhanunamikaze/AI-Dataset-Generator --skill dataset-generator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataset-generator
Source: https://github.com/Bhanunamikaze/AI-Dataset-Generator/tree/main
Command: npx skills add https://github.com/Bhanunamikaze/AI-Dataset-Generator --skill dataset-generator

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires jsonschema, requests, beautifulsoup4, trafilatura, duckduckgo-search, playwright, and includes scripts (resource) components.

What problem does it solve?

It removes the manual work of turning topics, URLs, local files, and existing datasets into training-ready data for AI models.

Core Features & Use Cases

  • Dataset generation: Create SFT and DPO examples with coverage steering, grounding, and quality gates.
  • Dataset operations: Collect source material, normalize imported files, verify records, deduplicate near-duplicates, and audit corpus quality.
  • Export and packaging: Produce OpenAI, HuggingFace, CSV, JSONL, and custom flat exports with data cards for agent IDE workflows.

Quick Start

Ask the skill to generate a dataset for your topic, such as a 1000-example customer support dataset in OpenAI JSONL.

Frequently Asked Questions about dataset-generator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate SFT and DPO training datasets from web URLs and local files?▼

To generate SFT and DPO training datasets, you provide topics, URLs, or local files. The skill collects source material, normalizes data, applies grounding checks, and exports training-ready JSONL or CSV files.

What is the best way to deduplicate existing JSONL datasets before fine-tuning an AI model?▼

Deduplicating existing JSONL datasets involves importing your file to apply automated near-duplicate detection and corpus-quality auditing. This ensures clean, verified records for fine-tuning.

Can I export generated datasets directly to HuggingFace and OpenAI JSONL formats?▼

Yes, you can export generated datasets directly to HuggingFace, OpenAI JSONL, CSV, and custom flat formats. The export process includes packaging with data cards for agent IDE workflows.

Does dataset generation support resumable processing for large-scale web research tasks?▼

Yes, dataset generation supports deterministic local processing with SQLite-backed resumable state. This allows large-scale web research and collection tasks to pause and resume without losing progress.

How do coverage steering and corpus-quality gates work during dataset generation?▼

Coverage steering and corpus-quality gates work by applying custom schemas and verification checks during generation. This enforces grounding and filters out low-quality records before export.

Do I need Playwright and BeautifulSoup to scrape web data for training set creation?▼

You need Playwright and BeautifulSoup dependencies to scrape and extract web data for training set creation. These libraries gather source material from URLs during the collection phase.

Related Skills

Explore More Data & Analytics Skills→
freeman5860freeman5860
33

ingest

Import materials into raw/ subdirectories and register pending items in raw/_registry.md.

Community
Intermediate
nirukk52nirukk52
1

e2e-testing

Automate ScreenGraph end-to-end UI validation with Playwright and .env configuration.

Community
Intermediate
DaddyElonMusk69DaddyElonMusk69

duckduckgo-search

Searches the web via DuckDuckGo without an API key.

Community
Intermediate
patricio0312revpatricio0312rev
5

doc-to-vector-dataset-generator

Convert documents into chunked JSONL datasets for embeddings.

Community
Advanced