data-engineer

Design scalable data ingestion and storage infrastructures with Kafka and schema registry.

Updated Apr 4, 2026

Search Tags:#etl #kafka #data-ingestion #schema-registry #timeseries #retention-policy

Ready to run

npx skills add https://github.com/asalhamed/dev-agents --skill data-engineer-asalhamed

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/asalhamed/dev-agents/tree/main/data-engineer
Command: npx skills add https://github.com/asalhamed/dev-agents --skill data-engineer-asalhamed

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Data engineers battle to build scalable pipelines and storage architectures that reliably move data from diverse sources to trusted stores, while enforcing governance and retention.

Core Features & Use Cases

Ingestion design and orchestration (Kafka topics, partitions, DLQ, backpressure)
Storage strategy (timeseries vs relational vs object storage, lifecycle policies)
Schema management (schema registry, evolution, backward compatibility)
Retention and data governance (hot/warm/cold/delete tiers, data catalog)
Observability and validation (load testing, monitoring, data lineage)

Quick Start

Define a telemetry ingestion workflow by selecting a source, map to topics, configure storage and retention, and run validation tests.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design a scalable Kafka data ingestion pipeline with backpressure handling?▼

Design scalable Kafka data ingestion by mapping data sources to topics and partitions, configuring dead letter queues (DLQs), and implementing backpressure handling to prevent system overload. Validate the workflow end-to-end before deployment.

What is the best way to manage schema registry compatibility for streaming data?▼

Manage schema registry compatibility by enforcing schema evolution rules and ensuring backward compatibility for streaming data. This prevents downstream consumers from breaking when upstream data structures change over time.

How do I configure data retention policies and storage tiers for timeseries telemetry?▼

Configure data retention by defining hot, warm, cold, and delete lifecycle tiers based on access patterns. Match timeseries telemetry to appropriate storage strategies to balance query performance and storage costs.

Can I use this approach for both batch and streaming data ingestion architectures?▼

Yes, this architecture handles both batch and streaming data ingestion. It designs scalable workflows applicable to diverse sources like IoT telemetry, enforcing governance and validation across both real-time and batch processing pipelines.

What is a dead letter queue (DLQ) and when do I need it for ETL workflows?▼

A dead letter queue (DLQ) isolates failed or malformed records during ETL workflows. You need it to prevent data loss and pipeline stalls when ingestion encounters invalid data, allowing safe reprocessing of failed messages.

How do I validate data lineage and governance across an IoT data catalog?▼

Validate data lineage and governance by implementing end-to-end monitoring and load testing across the IoT data catalog. This ensures trusted storage and tracks data origin and transformations throughout the pipeline.