sota-data-engineering
CommunityBuild and audit data pipelines with state-of-the-art rules.
Software Engineering#data quality#data pipeline#data engineering#performance tuning#CDC#warehouse#lakehouse
Authormartinholovsky
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill provides a comprehensive set of rules and guidelines for building and auditing data pipelines, ensuring best practices in data engineering.
Core Features & Use Cases
- Architecture and Modeling: Rules for designing data pipelines, including ELT vs. ETL, warehouse vs. lakehouse, and dimensional modeling.
- Pipeline and Orchestration: Guidelines for ensuring idempotency, incremental processing, and efficient orchestration.
- Streaming and CDC: Best practices for Kafka, Flink, and CDC patterns, including exactly-once processing and schema evolution.
- Data Quality and Contracts: Rules for data quality checks, including freshness, volume, uniqueness, and referential integrity.
- Storage and Performance: Guidelines for optimizing storage formats, partitioning, clustering, and performance tuning.
- Operations and Governance: Best practices for data environment isolation, deployment processes, access control, and observability.
Quick Start
Activate the sota-data-engineering skill and provide the relevant pipeline or warehouse for review.
Dependency Matrix
Required Modules
None requiredComponents
scriptsreferencesassets
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: sota-data-engineering Download link: https://github.com/martinholovsky/SOTA-skills/archive/main.zip#sota-data-engineering Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.