spark-pipeline

What problem does it solve?

Spark pipelines often fail in production due to missing schema enforcement, lack of auditability, and brittle error handling. This skill provides a structured approach to building reliable data pipelines with explicit schemas, audit columns, and robust validation.

Core Features & Use Cases

• Explicit schema enforcement and strong data quality checks in PySpark pipelines.
• Medallion architecture support (Bronze → Silver → Gold) with partitioned, idempotent writes.
• Use Cases: batch and streaming ETL for large datasets requiring reliable production-grade processing.

Quick Start

Set up a Spark Bronze ingestion pipeline with explicit schema and audit columns to write data to the Bronze layer.