golden-dataset-curation

Automating golden dataset curation involves running multi-agent evaluations on document accuracy, coherence, and relevance, then aggregating scores into a consensus-driven inclusion decision. This process handles ingestion, classification, and test-query generation to produce publish-ready entries.

Yes, Langfuse tracing is integrated directly into the curation workflow to monitor multi-agent consensus decisions. It captures the evaluation prompts, JSON-based scoring, and aggregation steps, ensuring the publish-ready consensus is fully auditable and traceable.

Multi-agent quality assessment works by running parallel agent evaluations on incoming documents to score accuracy, coherence, depth, and relevance. The agents aggregate these JSON-based scores to stratify difficulty, tag domains, and reach a consensus-driven inclusion decision.

The best way to classify content types during document ingestion is to use automated categorization that identifies formats like articles, tutorials, and research papers. This classification directly guides the subsequent curation, difficulty stratification, and test-query generation steps.

Manual review is not strictly required when inclusion thresholds are met, as the workflow commits publish-ready entries automatically. However, the consensus outputs and Langfuse traces can be used to guide manual review for borderline documents that fail to meet quality thresholds.