data-ingestion-builder

Create a new data ingestion provider by setting up a registry-driven loader in `src/ingest/{provider}/` with `registry.py`, `loader.py`, and tests following established templates. Configure registry mappings to connect your data source—APIs, files, or databases—to the pipeline and output Parquet files with metadata sidecars.

Use the provider scaffold pattern: organize loader code with storage abstraction, primary key validation tests, and sampling tools. This production-ready package structure ensures your ETL follows registry patterns and deploys consistently across environments with validated data integrity.

Write primary key tests and sampling validation following the provided templates. Test local and cloud deployment paths using the metadata sidecars and sample scaffolding tools included in the registry-driven design to catch data quality issues early.

Yes. The registry-driven approach supports adding multiple providers by repeating the scaffold pattern for each data source. Each provider package maintains its own loader, registry mappings, and tests while contributing Parquet outputs to a unified data pipeline.

Data ingestion providers write Parquet output files with metadata sidecars. This format enables efficient storage, schema validation through registry mappings, and integration with analytics workflows while maintaining data lineage and quality metadata.

The templates and guidance automate most registry design decisions. However, familiarity with registry concepts—mapping data sources to structured schemas—helps you configure providers effectively and troubleshoot custom data source requirements.