data-lake-architect

Data lake architecture organizes raw, processed, and curated data tiers with clear lineage. Use date-based or multi-dimensional partitioning to structure ingested data, apply Parquet or Iceberg formats for efficient storage, and implement lifecycle policies to manage retention and cost across cloud storage.

Partitioning strategies include time-based (by date), multi-dimensional (by business keys), and hashing approaches. Iceberg supports flexible partitioning evolution, and Parquet enables columnar compression. Choose based on query patterns and retention requirements to optimize performance and storage efficiency.

Organize storage in three tiers: raw (ingested data), processed (transformed), and curated (analytics-ready). Apply tiered retention policies to move data to cheaper cloud storage as it ages, balance accessibility with cost, and use Iceberg for schema evolution and lifecycle management.

Wide tables denormalize data for fast analytical queries but increase storage; normalized designs reduce redundancy but require joins. Data lakes support schema evolution strategies to shift between approaches. Choose based on query frequency, latency requirements, and storage budget.

Yes, Iceberg works with Parquet to enable advanced partitioning, hidden partitioning, and schema evolution without rewriting data. This combination provides better performance than traditional partitioning and simplifies lakehouse management on cloud storage.

A three-tier model (raw, processed, curated) clarifies data lineage and governance even at small scale, but complexity should match your retention policies and team size. Start with raw and curated tiers if budget-constrained, then add processed as analytics maturity grows.