skills.rest
ancoleman
ancolemanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ingesting-data

Facilitates data ingestion from cloud, APIs, files, and streaming sources into databases.

503|73|Updated Nov 13, 2025
Search Tags:#api#data pipeline#streaming#data ingestion#etl#cloud storage#cdc
One-click install
npx skills add https://github.com/ancoleman/ai-design-components --skill ingesting-data
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ingesting-data
Source: https://github.com/ancoleman/ai-design-components/tree/main/skills/ingesting-data
Command: npx skills add https://github.com/ancoleman/ai-design-components --skill ingesting-data

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires dlt, polars, boto3, confluent-kafka, requests, google-cloud-storage, azure-storage-blob, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill solves the challenge of efficiently and reliably importing data into your systems from diverse sources like cloud storage, APIs, files, and streaming platforms.

Core Features & Use Cases

  • Batch Ingestion: Load data from files (CSV, JSON, Parquet) or cloud storage (S3, GCS).
  • Streaming Ingestion: Process real-time data from Kafka or Kinesis.
  • API Data Feeds: Poll external APIs for updates or receive webhook events.
  • Change Data Capture (CDC): Replicate database changes.
  • Use Case: Automatically ingest daily sales reports from an S3 bucket, transform them, and load them into a PostgreSQL data warehouse.

Quick Start

Use the ingesting-data skill to load a CSV file named 'sales_data.csv' from the local directory into the 'sales' table in your PostgreSQL database.

Frequently Asked Questions about ingesting-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a data ingestion pipeline for files stored in S3 cloud storage?▼

To build a data ingestion pipeline for S3, you can load files in batch mode using the boto3 library to fetch data, then apply polars for transformation before loading it into your target database or data warehouse.

What is the best way to ingest real-time streaming data from Kafka?▼

The best way to ingest real-time streaming data from Kafka is by implementing a streaming ingestion pattern, which uses the confluent-kafka library to process and load continuous data feeds into your system.

Can I use this to replicate database changes using Change Data Capture?▼

Yes, you can use this to replicate database changes by implementing the Change Data Capture (CDC) pattern, which captures row-level database changes and loads them into your target ETL or ELT pipeline.

Does the data ingestion process support polling external APIs for updates?▼

The data ingestion process supports polling external APIs for updates by using the requests library to fetch data feeds, allowing you to routinely extract and load information into your database.

What libraries are needed to load CSV files into a PostgreSQL data warehouse?▼

To load CSV files into a PostgreSQL data warehouse, you need libraries like dlt for data loading, polars for transformation, and requests or boto3 if the files are fetched from external APIs or cloud storage.

Related Skills

Explore More Software Engineering Skills→
EntroVyxEntroVyx
3

hunt-cloud-misconfig

Detect cloud misconfigurations across AWS, GCP, and Azure environments.

Community
Advanced
Oruga420Oruga420

clickhouse-io

Optimize ClickHouse schemas, queries, and ingestion patterns for analytical workloads.

Community
Intermediate
stevenke1981stevenke1981
1

py-data

Build and review Python data workflows with pandas, Polars, Arrow, and DuckDB.

Community
Advanced
ai-enhanced-engineerai-enhanced-engineer
3

gcp-cicd-patterns

Automate CI/CD pipeline design on Google Cloud Platform with Cloud Build and Cloud Deploy.

Official
Advanced