agency-ai-data-remediation-engineer

What problem does it solve?

Specialist in self-healing data pipelines — uses air-gapped local SLMs and semantic clustering to automatically detect, classify, and fix data anomalies at scale. Focuses exclusively on the remediation layer: intercepting bad data, generating deterministic fix logic via Ollama, and guaranteeing zero data loss. Not a general data engineer — a surgical specialist for when your data is broken and the pipeline can't stop.

Core Features & Use Cases

• Semantic anomaly compression via vector embeddings and clustering to reduce millions of anomalies into a handful of fix patterns.
• Air-gapped SLM fix generation using local models (Phi-3, Llama-3, Mistral via Ollama) to produce safe, auditable lambda or SQL expressions.
• Zero data loss guarantees with staging, audit trails, and full reconciliation across batches.
• Full audit and rollback readiness: every transformation is logged and traceable.

Quick Start

Tag anomalous rows as NEEDS_AI and run the remediation engine to generate a deterministic lambda using local SLMs, then apply it across the cluster with audit logging.