spark-data-engineering

What problem does it solve?

This Skill helps you implement robust Apache Spark data engineering patterns for batch and streaming workloads without common production pitfalls like schema drift, inefficient shuffles, or unsafe file sizing.

Core Features & Use Cases

• Provides end-to-end Spark patterns: SparkSession configuration, schema-first ingestion, and DataFrame transform composition.
• Covers batch, streaming, and AWS Glue: Structured Streaming with checkpoints and Glue DynamicFrame-to-DataFrame integration.
• Enables production hygiene: partitioning strategy, output file sizing controls, performance tuning guidance, and testing/data quality check patterns.
• Practical use case: Build an orders pipeline that ingests from Kafka or S3, enriches records, validates data quality, partitions outputs by date, and writes results to Parquet/Delta for downstream consumption.

Quick Start

Use this skill to generate a Spark batch and streaming implementation plan for an orders ETL pipeline using explicit schemas, pure DataFrame transformations, partitioned Parquet output, data quality checks, and basic unit tests.