What problem does it solve? Broken data at scale cannot be fixed row-by-row, and sending sensitive records to cloud LLMs violates PII compliance. This Skill compresses thousands of anomalous rows into a handful of semantic clusters, generates deterministic fix logic with air-gapped local models, and enforces a mathematical zero-data-loss guarantee on every batch. ## Core Features & Use Cases - Semantic Anomaly Compression: Embeds suspect rows with sentence-transformers and clusters them in ChromaDB or FAISS, turning 50,000 errors into roughly a dozen actionable pattern families. - Air-Gapped Fix Generation: Uses Ollama-hosted models (Phi-3, Llama-3, Mistral) to output strictly validated Python lambdas, with forbidden-term rejection and confidence thresholds routing low-certainty clusters to human quarantine. - Zero-Data-Loss Reconciliation: Enforces Source_Rows == Success_Rows + Quarantine_Rows on every batch, triggering a Sev-1 alert on any mismatch, plus a full audit log of every transformation. - Use Case: A pipeline flags 50,000 rows with inconsistent date formats. The Skill clusters them into 12 patterns, generates 12 validated lambdas locally, applies them vectorized via pandas, and reconciles every row before staging promotion. ## Quick Start Assign this worker a mission to remediate the anomalous rows tagged NEEDS_AI in the staging table and produce a full audit log with reconciliation results.
