data-engineering

Provides guidance on designing scalable data pipelines for real-time analytics.

17|5|Updated Feb 6, 2026

Search Tags:#data quality #ETL #data engineering #data pipelines #data storage #ELT

Ready to run

npx skills add https://github.com/Logos-Liber/Atlas-Agent-Teams --skill data-engineering-logos-liber

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/Logos-Liber/Atlas-Agent-Teams/tree/main/teams/data-science/skills/data-engineering
Command: npx skills add https://github.com/Logos-Liber/Atlas-Agent-Teams --skill data-engineering-logos-liber

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the complexities of managing data flow, ensuring data quality, and selecting appropriate storage solutions for various data needs.

Core Features & Use Cases

Data Pipeline Patterns: Supports batch, streaming, Lambda, and Kappa architectures.
ETL/ELT Best Practices: Guides on efficient data extraction, transformation, and loading.
Data Storage Options: Explains SQL, NoSQL, data lakes, and data warehouses.
Data Quality & Validation: Covers techniques for ensuring data accuracy, completeness, and consistency.
Use Case: Design and implement a scalable data pipeline for real-time analytics on user clickstream data.

Quick Start

Explain the differences between batch and streaming data processing.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is the difference between batch and streaming data pipeline architectures?▼

Batch data pipeline architectures process data in scheduled, discrete chunks, whereas streaming architectures process continuous data flows in real time. This Skill explains both patterns alongside Lambda and Kappa architectures for combining historical and real-time data processing.

How do I choose between SQL, NoSQL, data lakes, and data warehouses for data storage?▼

To choose between data storage solutions, evaluate your data structure and query needs: SQL handles structured relational data, NoSQL suits flexible schemas, data lakes store raw unstructured data, and data warehouses optimize structured data for analytics. This Skill details these distinctions to guide your selection.

What are the best practices for ensuring data quality and validation in ETL pipelines?▼

Best practices for ensuring data quality in ETL pipelines involve implementing validation checks for accuracy, completeness, and consistency throughout the data flow. This Skill provides techniques to maintain data quality dimensions across extraction, transformation, and loading stages.

When should I use ETL methodologies compared to ELT for data pipelines?▼

You should use ETL methodologies when complex transformations are needed before loading data into storage, whereas ELT loads raw data first and transforms it within the target system. This Skill guides efficient extraction, transformation, and loading strategies for both approaches.

Can I design a scalable data pipeline for real-time analytics on clickstream data?▼

Yes, you can design a scalable data pipeline for real-time analytics on user clickstream data by applying streaming data pipeline patterns. This Skill provides use cases and architectural guidance for processing continuous high-volume data flows efficiently.

How do Lambda and Kappa architectures differ for data engineering pipelines?▼

Lambda architectures use separate batch and streaming layers for data processing, while Kappa architectures simplify data pipelines by using a single streaming layer for both real-time and historical processing. This Skill outlines when to apply each pattern for your data infrastructure.