skills.rest
mk-knight23
mk-knight23COMMUNITY
Explore all skills by this author →
Website𝕏@mk_knight_23GitHub

data-pipeline

Build and run Python ETL pipelines for CSV, JSON, Parquet, and database sources.

1|Updated Mar 4, 2026
Search Tags:#pandas#etl#polars#postgres#data-pipeline#data-cleaning#schema-validation
One-click install
npx skills add https://github.com/mk-knight23/AI-Agent-Nanobot --skill data-pipeline-mk-knight23
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-pipeline
Source: https://github.com/mk-knight23/AI-Agent-Nanobot/tree/main/skills/data-pipeline
Command: npx skills add https://github.com/mk-knight23/AI-Agent-Nanobot --skill data-pipeline-mk-knight23

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates the repetitive and error-prone work of building, running, and validating ETL pipelines to transform data between CSV, JSON, Parquet, and database sources so teams can move and clean data reliably without hand-writing boilerplate scripts.

Core Features & Use Cases

  • Profiling and inference: Samples inputs to infer schema, null rates, and cardinality so transformations are based on real data statistics.
  • Code generation and execution: Produces standalone Python pipeline_<name>.py scripts using pandas or polars (chosen by data size) and runs them with progress tracking.
  • Validation and reporting: Validates output schema, null rates, and row counts, and writes a PIPELINE_REPORT.md with before/after schemas and performance metrics.
  • Use case: Deduplicate and join orders from a Postgres table, normalize fields, and export a validated Parquet dataset for downstream analytics.

Quick Start

Run the data-pipeline skill to parse signup_date as UTC datetime, normalize emails to lowercase, drop rows with null revenue, and output the result as Parquet.

Frequently Asked Questions about data-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an ETL pipeline to clean and migrate CSV data to a database?▼

Build ETL pipelines to clean and migrate data by connecting to local files or databases. The pipeline profiles inputs, generates standalone Python scripts, validates output schemas, and produces a performance report.

Can I use pandas or polars to generate a data pipeline for Parquet files?▼

You can generate data pipelines using pandas or polars. The tool selects the library based on data size, profiles the input to infer schema, and executes the pipeline to process Parquet files.

Does this tool support schema validation and null rate checks for Postgres sources?▼

Schema validation and null rate checks are supported for Postgres connections. The pipeline validates output schemas, checks null rates and row counts, and writes results into a structured report file.

What is the best way to automate deduplication and format conversion for JSON datasets?▼

Automate deduplication and format conversion by generating a Python pipeline that profiles JSON inputs, normalizes fields, drops invalid rows, and exports validated results to formats like Parquet.

How do I validate output schemas after migrating data from SQLite to Parquet?▼

Validate output schemas after migration by running the generated pipeline, which automatically checks row counts, null rates, and schema integrity, then summarizes before and after statistics in a report.

Are there limitations when using polars for large-scale database migration workflows?▼

Polars is chosen for larger data sizes to optimize performance, but pipeline execution depends on local memory and database connection stability to process and validate the migration successfully.

Related Skills

Explore More Data & Analytics Skills→
brainbrewlabsbrainbrewlabs
7

data-collection

Collect data from APIs, databases, and files into validated CSV, JSON, or Parquet outputs.

Official
Intermediate
starlake-aistarlake-ai
209

parquet2csv

Convert Parquet files into CSV format for batch ETL pipelines.

Official
Intermediate
RxFitRxFit

data-autocleaning

Profile and clean BigQuery and GCS data using Dataplex results.

Community
Advanced
silvainfmsilvainfm
2

duckdb

Run SQL analytics directly on CSV, Parquet, and JSON files and Pandas or Polars DataFrames.

Community
Advanced