project-architecture

Medallion architecture organizes data processing into Bronze, Silver, and Gold layers. Bronze ingests raw data from sources like FVMS/CMS/NicheRMS, Silver cleanses and validates it, and Gold prepares analytics-ready datasets. Each layer has specific responsibilities for data quality, lineage tracking, and storage targets.

Azure ADLS Gen2 stores data across Bronze, Silver, and Gold layers. Azure Key Vault manages authentication and secrets. Azure Synapse handles analytics workflows. These services work together to support environment detection, configuration schemas, and end-to-end ingestion and transformation pipelines in both local and cloud contexts.

Error handling strategies include utilities like `NotebookLogger` for tracking issues and `TableUtilities` for consistent transformations. Implement logging at each layer to capture data lineage, validate transformations, and monitor both local and cloud ingestion processes. Document error patterns for debugging and optimization.

Yes. The medallion architecture scales across FVMS, CMS, and NicheRMS data sources simultaneously. Each source flows through Bronze, Silver, and Gold layers with unified ingestion, validation, and transformation workflows. Storage targets and authentication patterns remain consistent across all sources.

Core utilities include `SparkOptimiser` for query performance, `NotebookLogger` for observability, and `TableUtilities` for standardized transformations. These components enforce consistent patterns across ingestion, validation, and analytics workflows, reducing configuration overhead and improving maintainability.

Structure configuration schemas to define layer responsibilities, data targets, and authentication rules. Implement testing strategies that validate data lineage at each layer, verify transformation correctness, and confirm end-to-end workflow integrity across Bronze, Silver, and Gold stages in both local and production environments.