skills.rest
htlin222
htlin222COMMUNITY
Explore all skills by this author →
Website𝕏@htlin222GitHub

data-engineer

Designs and implements scalable data pipelines, warehouses, and streaming architectures.

77|4|Updated Jun 20, 2022
Search Tags:#data pipeline#streaming#data engineering#etl#airflow#data warehouse#spark
One-click install
npx skills add https://github.com/htlin222/dotfiles --skill data-engineer-htlin222
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/htlin222/dotfiles/tree/main/claude.symlink/skills/data-engineer
Command: npx skills add https://github.com/htlin222/dotfiles --skill data-engineer-htlin222

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the creation and management of scalable data pipelines, data warehouses, and streaming architectures, streamlining data processing and analytics.

Core Features & Use Cases

  • ETL/ELT Pipeline Design: Define and orchestrate data extraction, transformation, and loading processes.
  • Data Warehousing: Design and implement star schemas for efficient analytical querying.
  • Streaming Data Processing: Build real-time data ingestion and processing systems.
  • Data Quality Monitoring: Implement checks and validations to ensure data integrity.
  • Use Case: Design an ETL pipeline to ingest daily sales data from multiple sources, transform it into a star schema in a data warehouse, and monitor data quality.

Quick Start

Use the data-engineer skill to create an Airflow DAG for daily ETL processing.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an ETL pipeline using Airflow for daily data ingestion?▼

To design an ETL pipeline using Airflow, define a Directed Acyclic Graph (DAG) that orchestrates data extraction, transformation, and loading processes for daily data ingestion into your target system.

What is the best way to model a data warehouse with star schemas for analytics?▼

Modeling a data warehouse with star schemas involves designing fact and dimension tables to structure data efficiently, enabling fast and simplified analytical querying for business intelligence.

Can I use Spark for streaming data processing and real-time ingestion?▼

Yes, Spark can be utilized for streaming data processing to build real-time data ingestion systems, handling continuous data flows and enabling immediate data processing and analytics.

How do I monitor data quality and integrity in a data pipeline?▼

Monitor data quality by implementing checks and validations using tools like Great Expectations, ensuring data integrity by detecting anomalies and enforcing schema rules within your pipeline.

Does this approach support both ETL and ELT processes for data transformation?▼

Yes, this approach supports both ETL and ELT processes, allowing you to flexibly define data extraction, transformation, and loading sequences based on your specific data warehouse architecture.

Related Skills

Explore More Data & Analytics Skills→
boettiger-labboettiger-lab
4

cng-datasets

Verify geospatial data sources and orchestrate Kubernetes dataset workflows.

Official
Advanced
weirdgmeweirdgme
4

platform-engineering

Design and implement scalable platform engineering patterns for internal developer platforms.

Community
Advanced
pluginagentmarketplacepluginagentmarketplace
5

big-data

Process and analyze petabyte-scale data with Spark and Hadoop.

Community
Advanced
arm2armarm2arm
4

modal-serverless-gpu

Deploy and manage GPU workloads on Modal with serverless compute.

Community
Advanced