data-layer-architecture

To preserve AI extraction history while protecting manually verified data, implement a Bronze-Gold two-layer architecture. The Bronze layer stores immutable append-only extraction logs, while the Gold layer updates only when is_manually_verified is false, safeguarding human corrections during data curation.

The Bronze-Gold two-layer architecture is a data governance design that separates immutable AI extraction logs in the Bronze layer from confirmed data in the Gold layer. This structure enables full traceability, model auditing, and rollback capabilities while preventing automated updates from overwriting human-curated records.

You set up LLM traceability by creating an append-only ExtractionLog that captures every AI extraction immutably. Linking this log to Gold entities via a latest_extraction_log field allows you to audit extraction history, trace model outputs, and roll back confirmed data when necessary.

You can update Gold layer records without overwriting manually verified fields by checking the is_manually_verified flag. When the flag is true, the Gold entity is preserved; when false, new extractions update the Gold data, maintaining data integrity and compliance during automated curation.

Yes, this data governance approach works for tracking extractions across multiple entity types. The architecture enforces latest_extraction_log linkage across statements, politicians, speakers, and related entities, ensuring consistent traceability and auditability throughout the entire data curation workflow.