python-data-pipelines

Build a Python data pipeline by applying reusable ETL patterns that extract, transform, and load data using DLT, dbt, and Prefect 2 for orchestration. This approach provides standardized tooling conventions to accelerate scalable ELT workflows.

Orchestrate ETL workflows by leveraging pattern-driven Python pipelines that integrate DLT for extraction and loading with Prefect 2 for scheduling. This combination centralizes configuration and automates data pipeline deployment.

Yes, you can use uv and pyproject.toml to manage Python data engineering dependencies. The Skill establishes tooling conventions using uv for dependency management and pyproject.toml for centralizing pipeline configuration.

Integrate ML and LLM models into a Python data pipeline by applying reusable pipeline patterns designed for data engineering. These patterns support ML and LLM integrations alongside standard extraction, transformation, and orchestration tasks.

Use ruff for linting and ty for testing within your Python data pipelines. The Skill defines tooling conventions that include ruff and ty to ensure robust code quality and automated testing across ETL workflows.

You need a target data warehouse to deploy these Python ETL patterns effectively. The provided templates demonstrate building end-to-end pipelines from data sources like Prometheus directly to a warehouse with automated testing.