skills.rest
TheGrowthExponent
TheGrowthExponentOFFICIAL
Explore all skills by this author →
WebsiteGitHub

Data Engineering Skills

Build and orchestrate Databricks ETL pipelines with Delta Live Tables.

Updated Feb 23, 2026
Search Tags:#workflow automation#data quality#etl#medallion architecture#pipeline orchestration#databricks#delta live tables
One-click install
npx skills add https://github.com/TheGrowthExponent/c7-databricks --skill data-engineering-skills
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Data Engineering Skills
Source: https://github.com/TheGrowthExponent/c7-databricks/tree/main/docs/skills/data-engineering
Command: npx skills add https://github.com/TheGrowthExponent/c7-databricks --skill data-engineering-skills

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the friction of building production-grade data engineering workflows on Databricks by providing consolidated, actionable guidance for ETL pipelines, job orchestration, and Delta Live Tables, removing the need to search across fragmented official documentation for common implementation patterns.

Core Features & Use Cases

  • Databricks Jobs Orchestration: Covers multi-task workflow creation, scheduling, error handling, retries, and monitoring for batch and streaming ETL pipelines.
  • Delta Live Tables (DLT) Implementation: Provides Python and SQL examples for declarative pipeline definition, data quality expectations, CDC processing, and medallion architecture (Bronze/Silver/Gold) design.
  • Production-Ready Patterns: Includes idempotent merge logic, incremental loading with watermarking, synthetic data generation for testing, and best practices for resource management and cost optimization. Use Case: For example, a data engineer can use this Skill to quickly build a daily sales processing pipeline that ingests raw data, validates quality, aggregates business metrics, and sends failure alerts, all following Databricks recommended patterns.

Quick Start

Use the data-engineering skill to create a scheduled multi-task Databricks job for daily ETL processing with error handling and email notifications.

Frequently Asked Questions about Data Engineering Skills

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a Databricks ETL pipeline with Delta Live Tables?▼

To build a Databricks ETL pipeline with Delta Live Tables, you define declarative data pipelines using Python or SQL, applying data quality expectations and CDC processing to structure your medallion architecture workflows.

What is the best way to orchestrate multi-task Databricks jobs?▼

The best way to orchestrate multi-task Databricks jobs is by configuring scheduled workflows with built-in error handling, retries, and monitoring for batch and streaming ETL pipelines.

How do I implement incremental data loading and idempotent merges in Databricks?▼

You implement incremental data loading and idempotent merges in Databricks by applying watermarking patterns and specific merge logic that safely processes CDC data without duplicating records.

How do I enforce data quality constraints in a Databricks medallion architecture?▼

You enforce data quality constraints in a Databricks medallion architecture by defining expectations within your declarative Delta Live Tables pipelines to validate records moving through the Bronze, Silver, and Gold layers.

Can I generate synthetic data for testing Databricks ETL pipelines?▼

Yes, you can generate synthetic data for testing Databricks ETL pipelines to validate workflow logic and data quality expectations before deploying scheduled multi-task jobs to production environments.

How do I set up monitoring and alerting for production data workloads in Databricks?▼

You set up monitoring and alerting for production data workloads in Databricks by configuring failure alerts and email notifications directly within your multi-task job orchestration setup to track pipeline health.

Related Skills

Explore More Data & Analytics Skills→
alexeybanalexeyban
21

databricks-dq-automation

Automate post-change data quality validation for Databricks SQL outputs.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-medallion-architecture

Design StarRocks Bronze, Silver, and Gold medallion schemas for CDC pipelines.

Community
Advanced
Zack2626-okZack2626-ok

databricks-app-apx

Automate setup and management of full-stack Databricks Apps with React and FastAPI.

Community
Advanced
hiddink-aihiddink-ai
1

pipeline-architecture-patterns

Guide data pipeline architecture with ETL/ELT, orchestration, and data quality patterns.

Official
Advanced