skills.rest
eaglhuang
eaglhuangCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

general-data-pipeline

Automate collection, mapping, deduplication, classification, validation, and export of generals data.

3|Updated Mar 6, 2026
Search Tags:#validation#deduplication#export#data-pipeline#web-scrape#data-mapping#generals
One-click install
npx skills add https://github.com/eaglhuang/cocos_new --skill general-data-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: general-data-pipeline
Source: https://github.com/eaglhuang/cocos_new/tree/main/.github/skills/general-data-pipeline
Command: npx skills add https://github.com/eaglhuang/cocos_new --skill general-data-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill provides an end-to-end pipeline to collect, map, deduplicate, classify, validate, and export generals data from multiple sources to a structured master dataset.

Core Features & Use Cases

  • End-to-end data pipeline from public sources to a master dataset for generals.
  • Phase-driven workflow with scrape, map, merge, classify, validate, and export stages; supports scope controls (faction, name list, all) and sources (wiki, koei, manual).
  • Built-in quality checks, deduplication, and layered outputs (base, lore, stories) for batch imports.

Quick Start

Ingest and normalize a batch of generals data from wiki/koei sources and export it to the master dataset.

Frequently Asked Questions about general-data-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate end-to-end data collection and deduplication for batch generals onboarding?▼

Automate batch generals onboarding by running a phase-driven data pipeline that scrapes wiki and koei sources, maps fields, deduplicates entries, and outputs a structured master dataset with unique ids.

What is the best way to build a master dataset of generals from multiple wiki and koei sources?▼

Build a master dataset by executing pipeline phases to scrape, merge, and classify generals data, applying tiered outputs like base, lore, and stories to structure the final reference values.

How do I validate data integrity and generate unique ids when importing hundreds of generals?▼

Validate imported generals data integrity by running the validation pipeline phase, which executes built-in quality checks and generates unique ids to prevent duplication across the master dataset.

Can I run specific data mapping and classification phases instead of a full pipeline scrape?▼

Yes, you can run specific phases like map, merge, or classify individually, or execute a full pipeline run, applying scope controls to target specific factions or name lists during data mapping.

Does the generals data pipeline support manual data sources alongside automated web scraping?▼

Yes, the pipeline supports manual data sources alongside automated wiki and koei scraping, allowing you to merge manually entered generals data into the same classification and validation workflow.

Related Skills

Explore More Data & Analytics Skills→
salmanabdurrahmansalmanabdurrahman

recon-js-endpoints

Normalize, deduplicate, and categorize API endpoints from JavaScript files.

Community
Advanced
alfredxwalfredxw
587

lore

Read, update, delete, and query lore items in a text adventure game database.

Community
Intermediate
LuminairPrimeLuminairPrime

mcp-firecrawl

Scrape web pages and extract structured content using Firecrawl.

Community
Advanced
thejefflarsonthejefflarson
20

training-data-poisoning

Block poisoned or low-quality data from training and fine-tuning pipelines.

Community
Advanced