skills.rest
dtsong
dtsongCOMMUNITY
Explore all skills by this author →
GitHub

python-data-engineering

Write Python data pipelines with Polars, Pandas, PySpark, and dbt models.

Updated Aug 23, 2026
Search Tags:#python#data validation#data engineering#pandas#polars#pyspark#api extraction
One-click install
npx skills add https://github.com/dtsong/data-engineering-skills --skill python-data-engineering-dtsong
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: python-data-engineering
Source: https://github.com/dtsong/data-engineering-skills/tree/main/python-data-engineering
Command: npx skills add https://github.com/dtsong/data-engineering-skills --skill python-data-engineering-dtsong

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides expert guidance for writing robust, efficient, and maintainable Python code for data engineering tasks, covering everything from data manipulation and validation to API extraction and distributed processing.

Core Features & Use Cases

  • DataFrame Manipulation: Write high-performance data transformations using Polars, Pandas, and PySpark.
  • Data Validation: Implement data quality checks with Pydantic, Pandera, and Great Expectations.
  • API Extraction: Build reliable scripts to fetch data from external APIs.
  • dbt Python Models: Integrate Python logic directly into your dbt projects.
  • Use Case: You need to process a large CSV file, clean the data using Pandas, validate it with Pandera, and then write the results to a Parquet file.

Quick Start

Use the python-data-engineering skill to write a Polars script that reads 'input.csv', filters rows where 'amount' is greater than 100, and saves the result to 'output.parquet'.

Frequently Asked Questions about python-data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I write a Python data pipeline to extract API data and validate it?▼

Build a Python data pipeline by using this skill to generate reliable API extraction scripts and implement data quality checks with Pydantic, Pandera, or Great Expectations.

What is the best way to process large CSV files in Python and export to Parquet?▼

Process large CSV files in Python and export to Parquet by using DataFrame libraries like Polars or PySpark for high-performance transformations, leveraging efficient and immutable data processing patterns.

Does this Python data engineering guidance work with dbt Python models?▼

Yes, this Python data engineering guidance supports dbt Python models by providing expert code generation to integrate Python logic directly into your dbt projects for robust data transformations.

How do I validate a Pandas DataFrame in a data pipeline?▼

Validate a Pandas DataFrame in a data pipeline by implementing automated data quality checks using Pandera or Great Expectations, ensuring your data meets type-safe schemas before downstream processing.

Should I use Polars or Pandas for Python data transformations?▼

Choose Polars over Pandas for Python data transformations when you need high-performance, immutable data processing patterns, while Pandas remains suitable for standard DataFrame manipulations within your pipeline.

Related Skills

Explore More Software Engineering Skills→
Wike-CHIWike-CHI
2

data-automation-service

Automate data cleaning and transformation for raw datasets.

Community
Intermediate
Alex1980AlexAlex1980Alex

duckdb-analytics

Analyze JSONL and Parquet logs directly with DuckDB.

Community
Advanced
AubreyHanAubreyHan

gcp-pipeline-orchestration

Generate and update Google Cloud Composer orchestration YAML for data pipelines.

Community
Advanced
nksethnkseth

spark-engineer

Optimize Spark DataFrame and RDD pipelines with performance tuning.

Community
Advanced