ingesting-into-data-lake

Create a Glue connection of type Snowflake first, then use a Glue 5.1+ PySpark job that reads via the connection and writes with writeTo() to the S3 Tables catalog. The job needs the --datalake-formats iceberg argument and catalog config passed through --conf.

For tables under roughly 100 GB, use Athena CTAS targeting s3tablescatalog with optional Iceberg partition transforms. For larger tables or complex transforms, use a Glue ETL PySpark job that reads from the Glue Data Catalog and writes with createOrReplace().

No, DynamoDB does not need a Glue connection. Small tables can be read directly by Glue using the dynamodb connection type, while medium and large tables should use native export-table-to-point-in-time to S3, which consumes no read capacity.

This error occurs when Spark catalog configuration is set via spark.conf.set() inside the script. In Glue 5.x, all spark.sql.catalog.* settings must be passed in the --conf job argument instead, and S3 Tables configs must include the warehouse parameter.

SaaS and streaming sources such as Salesforce, ServiceNow, SAP, MongoDB, and Kafka are not supported by this ingestion workflow. Connection setup and troubleshooting are also out of scope and handled by a separate connection skill.

Athena CTAS works for tables under about 100 GB but has a 100-partition limit per statement and a 30-minute timeout. Glue ETL handles larger tables, complex PySpark transforms, and scheduled repeatable migrations with configurable worker counts.