skills.rest
mreferre
mreferreCOMMUNITY
Explore all skills by this author →
Website𝕏@mreferreGitHub

ingesting-into-data-lake

Import data from S3, JDBC, Redshift, Snowflake, BigQuery, DynamoDB, or Glue tables into AWS data lake tables.

Updated May 12, 2026
Search Tags:#data ingestion#iceberg#s3 tables#glue etl#jdbc migration#etl scheduling#data lake validation
One-click install
npx skills add https://github.com/mreferre/aws-agent-toolkit-skills --skill ingesting-into-data-lake
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ingesting-into-data-lake
Source: https://github.com/mreferre/aws-agent-toolkit-skills/tree/main/.kiro/skills/ingesting-into-data-lake
Command: npx skills add https://github.com/mreferre/aws-agent-toolkit-skills --skill ingesting-into-data-lake

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates importing data from common sources (S3 files, local uploads, JDBC databases, Redshift, Snowflake, BigQuery, DynamoDB, and existing Glue catalog tables) into queryable AWS data lake tables so downstream analytics can run without manual ETL setup.

Core Features & Use Cases

  • Source-to-table ingestion with safe routing: Routes by source type (local files, S3 files, JDBC, Snowflake, BigQuery, DynamoDB, or catalog migration) and uses the right reference workflow.
  • Target format selection with posture-aware defaults: Defaults to S3 Tables (and recommends standard Iceberg on existing buckets if S3 Tables isn’t adopted), with support for one-time loads, recurring pipelines, and migrations.
  • Operational guardrails and validation: Verifies tooling availability, region/credentials, requires Glue connection verification for JDBC/Snowflake/BigQuery, and runs row-count + null checks + sample spot-check validation before proceeding.
  • Scheduling for recurring pipelines: Creates Glue Triggers for recurring single-step imports and delegates multi-step branching cases to MWAA.
  • Explicit guardrails for unsupported use cases: Declines SaaS/streaming sources (e.g., Salesforce, ServiceNow, SAP, MongoDB, Kafka) and delegates Glue connection setup/troubleshooting to the appropriate skill.

Quick Start

Import data from an existing S3 prefix into S3 Tables by running the ingest workflow for your source path and choosing the target format, for example: ingesting-into-data-lake with source-path s3://my-bucket/path/to/data and --target s3-tables.

Frequently Asked Questions about ingesting-into-data-lake

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest data from a JDBC database into an AWS data lake?▼

To ingest data from a JDBC database into an AWS data lake, the workflow verifies your Glue connection, extracts the source data, and loads it into queryable Iceberg or S3 Tables. It enforces connection verification before moving data and performs post-load validation checks.

What is the best way to migrate data from Snowflake or BigQuery to S3 Tables?▼

The best way to migrate data from Snowflake or BigQuery to S3 Tables is to use a catalog migration workflow that verifies your Glue connection, extracts the source data, and loads it into queryable AWS data lake tables. The process includes row-count checks and sample spot-check validation.

How do I set up recurring ETL pipelines for S3 files into an Iceberg data lake?▼

To set up recurring ETL pipelines for S3 files into an Iceberg data lake, the workflow creates Glue Triggers for single-step recurring imports. For multi-step branching pipelines, it delegates scheduling to MWAA to manage the recurring data ingestion.

Can I ingest data from streaming sources like Kafka or SaaS apps into S3 Tables?▼

No, you cannot ingest data from streaming sources like Kafka or SaaS apps like Salesforce and ServiceNow into S3 Tables using this workflow. It explicitly declines these sources and focuses on batch sources like S3 files, JDBC systems, DynamoDB, and data warehouses.

Do I need AWS MCP tools to import local files into a queryable data lake?▼

You need AWS MCP tools to import local files into a queryable data lake when available, with AWS CLI as a fallback. The workflow checks tooling availability and region credentials before routing the ingestion process to the appropriate target format.

What validation checks are performed after loading data into an AWS data lake?▼

After loading data into an AWS data lake, the workflow performs post-load validation including row counts, null checks, and spot samples. This ensures the ingested data matches the source and maintains quality before downstream analytics run.

Related Skills

Explore More Data & Analytics Skills→
awsaws
2.3k

ingesting-into-data-lake

Ingest data from S3, JDBC, Snowflake, BigQuery, DynamoDB, and Glue catalogs into AWS data lakes.

Official
Advanced
mreferremreferre

creating-data-lake-table

Create managed Iceberg tables on Amazon S3 Tables with Glue catalog registration.

Community
Advanced
AndreKuraitAndreKurait

ingesting-into-data-lake

Ingest data from diverse sources into AWS data lake targets.

Community
Advanced
wesleyosantos91wesleyosantos91
6

data-platform

Design AWS data platform architectures for ETL/ELT and data lake scenarios.

Community
Intermediate