bigdata-product-extension

What problem does it solve?

Big data changes can silently break downstream consumers, corrupt analytics, or cause runaway cost and operational failure when schema evolution, partitioning, idempotency, quality gates, lineage, and governance are not handled as a coordinated product change.

Core Features & Use Cases

• Schema evolution safety for distributed data products: Enforces backward-compatible changes across consumers using explicit compatibility intent and impact analysis.
• Pipeline reliability and replay safety: Ensures every pipeline stage is idempotent and re-runnable, preventing duplication and silent data loss after failures.
• Data quality, PII governance, and cost guardrails: Requires data quality gates before promotion, inventories PII for compliant erasure strategy, and mandates cost estimation for scan-heavy workloads.

Quick Start

Ask the agent to assess your proposed Spark/Flink/dbt/Kafka/data-lake change and return a blocked-or-approved big data readiness decision with schema, partition, idempotency, quality gate, PII erasure, cost, lineage, analytics, and MLOps requirements.