What problem does it solve? Raw data exports are messy: mixed encodings, duplicates, inconsistent timestamps, and malformed records. This Skill provides a repeatable three-stage pipeline that cleans data with CLI tools, enriches it with Python NLP, and integrates LLM scoring, with validation gates that prevent silent data loss between stages. ## Core Features & Use Cases - Staged ETL Architecture: Stage 1 uses CLI tools (csvkit, jq, minet, qsv) for cleaning and validation; Stage 2 uses Python (pandas, spaCy, NLTK, scikit-learn) for NLP enrichment; Stage 3 integrates LLM scoring and composite metrics. - Validation Gates and Manifests: Every stage reconciles row counts, quarantines failed records instead of dropping them, and writes JSON manifests as an audit trail. - Graceful Degradation: Handles small corpora, sparse metadata, mixed encodings, and LLM API failures with explicit fallback behavior and coverage reporting. - Use Case: You receive a 50,000-row Reddit data export as JSON. The pipeline deduplicates and normalizes it with jq and qsv, runs spaCy NER and VADER sentiment in batches, then applies LLM relevance scoring with checkpointing, producing an analysis-ready dataset and a pipeline report. ## Quick Start Use the tiered-processing-pipeline skill to process my raw data export in exports/ into a clean, enriched dataset with a pipeline report.
