ingesting-into-data-lake

What problem does it solve?

This Skill automates importing data from common sources (S3 files, local uploads, JDBC databases, Redshift, Snowflake, BigQuery, DynamoDB, and existing Glue catalog tables) into queryable AWS data lake tables so downstream analytics can run without manual ETL setup.

Core Features & Use Cases

• Source-to-table ingestion with safe routing: Routes by source type (local files, S3 files, JDBC, Snowflake, BigQuery, DynamoDB, or catalog migration) and uses the right reference workflow.
• Target format selection with posture-aware defaults: Defaults to S3 Tables (and recommends standard Iceberg on existing buckets if S3 Tables isn’t adopted), with support for one-time loads, recurring pipelines, and migrations.
• Operational guardrails and validation: Verifies tooling availability, region/credentials, requires Glue connection verification for JDBC/Snowflake/BigQuery, and runs row-count + null checks + sample spot-check validation before proceeding.
• Scheduling for recurring pipelines: Creates Glue Triggers for recurring single-step imports and delegates multi-step branching cases to MWAA.
• Explicit guardrails for unsupported use cases: Declines SaaS/streaming sources (e.g., Salesforce, ServiceNow, SAP, MongoDB, Kafka) and delegates Glue connection setup/troubleshooting to the appropriate skill.

Quick Start

Import data from an existing S3 prefix into S3 Tables by running the ingest workflow for your source path and choosing the target format, for example: ingesting-into-data-lake with source-path s3://my-bucket/path/to/data and --target s3-tables.