docker-databricks-lab-ops

You can automate a local Docker Databricks CDC lab setup using script-driven orchestration to start services, register the Debezium connector, generate CDC traffic bursts, and validate Bronze/Silver/Gold notebook outcomes in a repeatable environment.

Debezium CDC data generation captures database changes in controlled bursts and routes them through the local Docker stack to Databricks. The orchestration scripts trigger notebook jobs to process these events and validate the resulting data outputs across Bronze, Silver, and Gold tables.

To reset Databricks tables and trigger notebook jobs for verification, run the included scripts which orchestrate deterministic actions to clear existing data, execute notebook runs, and poll results to ensure end-to-end success of the CDC pipeline.

Yes, you need Docker to bring up the local CDC stack and the Databricks SDK along with Python dependencies to interact with the Databricks environment. These components are required to orchestrate services, generate traffic, and validate notebook outcomes.

Yes, you can validate Bronze, Silver, and Gold table progress automatically. The scripts orchestrate the pipeline by generating Debezium CDC events, triggering notebook jobs to process the data, and polling results to verify end-to-end success across all table layers.

The best way to run a repeatable Databricks CDC verification environment is using script-driven orchestration. This approach applies deterministic actions via scripts to start Docker services, generate traffic, reset tables, and validate outputs consistently across multiple runs.