add-datalake-consumer

You can ingest RabbitMQ events into Azure Data Lake by configuring a consumer that subscribes to queues, maps the payload to a Parquet schema, and writes partitioned files to Blob storage with buffering and batching for optimized throughput.

Partitioning Parquet event data by year, month, and day in Azure Data Lake enables efficient time-based queries for BI dashboards and ML workflows, allowing analytics engines to scan only relevant date partitions during retrieval.

Yes, this event ingestion pattern is designed for logging pipelines where risk.evaluated events must be stored in a scalable Data Lake, mapping specific Parquet columns to support downstream BI dashboards and ML outcome analysis.

You configure the RabbitMQ and data_lake settings in your default.yaml file, define the consumer class to map queue messages to the required Parquet schema, and start the server to run the DataLakeCollector continuously.

Yes, the event ingestion process supports buffering and batching to optimize write throughput and cost, accumulating RabbitMQ events before committing them as Parquet files to Azure Blob storage.

Parquet partitioning by year, month, and day requires strict adherence to documented schema layouts for risk events, meaning any unmapped columns or incorrect path configurations will break downstream BI dashboard and ML data access pipelines.