skills.rest
dtsong
dtsongCOMMUNITY
Explore all skills by this author →
GitHub

dlt-extract

Create portable DLT pipelines for file-based data extraction with destination swapping.

Updated Aug 27, 2026
Search Tags:#data pipeline#consulting#file extraction#dlt#portable pipeline#schema contract
One-click install
npx skills add https://github.com/dtsong/data-engineering-skills --skill dlt-extract
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dlt-extract
Source: https://github.com/dtsong/data-engineering-skills/tree/main/dlt-extract
Command: npx skills add https://github.com/dtsong/data-engineering-skills --skill dlt-extract

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the process of building portable DLT pipelines, specifically for extracting data from file-based sources and supporting consulting engagements where clients need to run pipelines themselves.

Core Features & Use Cases

  • File Source Ingestion: Handles CSV, Excel, Parquet, JSON, SharePoint, and SFTP files.
  • Destination Swapping: Enables pipelines to target DuckDB for development and Snowflake/BigQuery for production using environment variables.
  • Schema Contracts: Enforces data quality and handles schema drift.
  • Portable Pipelines: Facilitates client handoff with pip install and environment variable configurations.
  • Use Case: You need to ingest daily CSV reports from a client's SFTP server, process them locally with DuckDB, and then deploy the same pipeline to load data into Snowflake.

Quick Start

Use the dlt-extract skill to create a portable DLT pipeline that reads CSV files from a local directory and loads them into DuckDB.

Frequently Asked Questions about dlt-extract

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a portable data pipeline for extracting files from SFTP?▼

You can build a portable data pipeline using dlt to extract files from SFTP sources. The pipeline can be packaged for client handoff with pip install and configured using environment variables, allowing clients to run it independently.

Can I develop a DLT pipeline locally with DuckDB and deploy it to Snowflake?▼

Yes, DLT pipelines support destination swapping, enabling you to develop locally using DuckDB and deploy the same pipeline to Snowflake or BigQuery in production by changing environment variables.

What file formats can I ingest using a DLT pipeline for data extraction?▼

DLT pipelines support file source ingestion from CSV, Excel, Parquet, JSON, SharePoint, and SFTP files, allowing you to extract structured and semi-structured data for consulting and analytics use cases.

How do DLT schema contracts handle data quality and schema drift?▼

DLT schema contracts enforce data quality and manage schema drift during file extraction. This ensures your portable pipeline maintains consistent data structures even when upstream file formats change unexpectedly.

What is the best way to hand off a data pipeline to a client?▼

The best way to hand off a pipeline is creating a portable DLT pipeline configured with environment variables. Clients can simply run pip install and execute the pipeline to ingest their own file data.

Related Skills

Explore More Data & Analytics Skills→
dlt-hubdlt-hub
53

create-rest-api-pipeline

Generate a dlt pipeline for ingesting data from a REST API source.

Official
Intermediate
Amar1404Amar1404
1

connect-data

Connect to CSV, DuckDB, PostgreSQL, BigQuery, Snowflake, Athena, and ClickHouse data sources.

Community
Intermediate
levanminhduclevanminhduc

docker

Guide containerization of applications with Dockerfiles, Compose, and CI/CD pipelines.

Community
Advanced
GhostOf0daysGhostOf0days

alternative-data-pipeline

Automates alternative data pipeline workflows with schema contracts and lineage tracking.

Community
Advanced