Design scalable data ingestion and storage infrastructures with Kafka and schema registry.
npx skills add https://github.com/asalhamed/dev-agents --skill data-engineer-asalhamed
Please help me install this Agent Skill. Skill: data-engineer Source: https://github.com/asalhamed/dev-agents/tree/main/data-engineer Command: npx skills add https://github.com/asalhamed/dev-agents --skill data-engineer-asalhamed
Data engineers battle to build scalable pipelines and storage architectures that reliably move data from diverse sources to trusted stores, while enforcing governance and retention.
Define a telemetry ingestion workflow by selecting a source, map to topics, configure storage and retention, and run validation tests.
High-intent search queries and answers about installing and using this skill.
Design scalable Kafka data ingestion by mapping data sources to topics and partitions, configuring dead letter queues (DLQs), and implementing backpressure handling to prevent system overload. Validate the workflow end-to-end before deployment.
Manage schema registry compatibility by enforcing schema evolution rules and ensuring backward compatibility for streaming data. This prevents downstream consumers from breaking when upstream data structures change over time.
Configure data retention by defining hot, warm, cold, and delete lifecycle tiers based on access patterns. Match timeseries telemetry to appropriate storage strategies to balance query performance and storage costs.
Yes, this architecture handles both batch and streaming data ingestion. It designs scalable workflows applicable to diverse sources like IoT telemetry, enforcing governance and validation across both real-time and batch processing pipelines.
A dead letter queue (DLQ) isolates failed or malformed records during ETL workflows. You need it to prevent data loss and pipeline stalls when ingestion encounters invalid data, allowing safe reprocessing of failed messages.
Validate data lineage and governance by implementing end-to-end monitoring and load testing across the IoT data catalog. This ensures trusted storage and tracks data origin and transformations throughout the pipeline.
expanso-ioStream data from Kafka topics to S3 buckets with partitioning and batching.
JNZader-VaultDevelop Power BI solutions with DAX measures, Power Query transformations, and dashboard designs.
DKSangEvaluate data lineage, metadata stewardship, access control, and privacy handling for governance compliance.
1thirteeng3Deploy ML workloads to Modal's serverless GPU cloud with auto-scaling.
