general-data-pipeline

Automate collection, mapping, deduplication, classification, validation, and export of generals data.

3|Updated Mar 6, 2026

Search Tags:#validation #deduplication #export #data-pipeline #web-scrape #data-mapping #generals

Ready to run

npx skills add https://github.com/eaglhuang/cocos_new --skill general-data-pipeline

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: general-data-pipeline
Source: https://github.com/eaglhuang/cocos_new/tree/main/.github/skills/general-data-pipeline
Command: npx skills add https://github.com/eaglhuang/cocos_new --skill general-data-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill provides an end-to-end pipeline to collect, map, deduplicate, classify, validate, and export generals data from multiple sources to a structured master dataset.

Core Features & Use Cases

End-to-end data pipeline from public sources to a master dataset for generals.
Phase-driven workflow with scrape, map, merge, classify, validate, and export stages; supports scope controls (faction, name list, all) and sources (wiki, koei, manual).
Built-in quality checks, deduplication, and layered outputs (base, lore, stories) for batch imports.

Quick Start

Ingest and normalize a batch of generals data from wiki/koei sources and export it to the master dataset.

Frequently Asked Questions about general-data-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate end-to-end data collection and deduplication for batch generals onboarding?▼

Automate batch generals onboarding by running a phase-driven data pipeline that scrapes wiki and koei sources, maps fields, deduplicates entries, and outputs a structured master dataset with unique ids.

What is the best way to build a master dataset of generals from multiple wiki and koei sources?▼

Build a master dataset by executing pipeline phases to scrape, merge, and classify generals data, applying tiered outputs like base, lore, and stories to structure the final reference values.

How do I validate data integrity and generate unique ids when importing hundreds of generals?▼

Validate imported generals data integrity by running the validation pipeline phase, which executes built-in quality checks and generates unique ids to prevent duplication across the master dataset.

Can I run specific data mapping and classification phases instead of a full pipeline scrape?▼

Yes, you can run specific phases like map, merge, or classify individually, or execute a full pipeline run, applying scope controls to target specific factions or name lists during data mapping.

Does the generals data pipeline support manual data sources alongside automated web scraping?▼

Yes, the pipeline supports manual data sources alongside automated wiki and koei scraping, allowing you to merge manually entered generals data into the same classification and validation workflow.