data-quality

Enforces data integrity and provenance across multi-source extraction pipelines.

1|Updated Apr 30, 2026

Search Tags:#validation #qa #provenance #data-integrity #data-quality #sparql

Ready to run

npx skills add https://github.com/danbri/forgetmenot --skill data-quality-danbri

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-quality
Source: https://github.com/danbri/forgetmenot/tree/main/skills/data-quality
Command: npx skills add https://github.com/danbri/forgetmenot --skill data-quality-danbri

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires python3, and includes scripts (resource) components.

What problem does it solve?

This skill addresses the degradation of data corpora caused by noisy, multi-source ingestion, preventing the common pitfall of creating unreliable hybrid datasets that mask underlying source errors.

Core Features & Use Cases

Diagnostic Discipline: Provides a framework to categorize data gaps as crawl-misses, extractor-misses, or upstream source issues rather than applying superficial fixes.
Cross-Extractor Validation: Enforces independent extraction paths for the same source (e.g., HTML vs. API) to identify and resolve discrepancies.
Anchor-Based Testing: Implements a system of known-good test cases to ensure pipeline stability and provide fast, noisy failure alerts when data quality regresses.

Quick Start

Use the data-quality skill to audit the current extraction pipeline by running the established anchor test suite against the latest data.

Frequently Asked Questions about data-quality

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I validate data integrity during multi-source ingestion?▼

Data integrity during multi-source ingestion is validated through systematic cross-checking, anchor-case validation, and stable identifier usage to maintain high-confidence corpora. This framework prevents noisy ingestion from masking underlying source errors.

What is the best way to diagnose data extraction gaps from APIs?▼

Diagnosing data extraction gaps requires categorizing them as crawl-misses, extractor-misses, or upstream source issues. This diagnostic discipline prevents superficial fixes by addressing the actual root cause of the API extraction failure.

How do I set up cross-extractor validation for Wikidata and gov.uk datasets?▼

Cross-extractor validation enforces independent extraction paths for the same source, such as HTML versus API, to identify and resolve discrepancies. This ensures reliable data extraction across Wikidata and gov.uk datasets.

How does anchor-based testing prevent data quality regression in pipelines?▼

Anchor-based testing implements known-good test cases to ensure pipeline stability and provide fast, noisy failure alerts. This system quickly exposes data quality regression when run against the latest extracted data.

Can I use Python to audit data provenance across UK Parliament APIs?▼

Python is required to audit data provenance across UK Parliament APIs. The skill uses Python scripts to apply provenance standards and run established anchor test suites against complex data extraction tasks.

Why does my hybrid dataset from multiple sources contain masked errors?▼

Hybrid datasets contain masked errors due to the degradation caused by noisy, multi-source ingestion. Applying rigorous provenance standards and cross-checking prevents the creation of unreliable hybrid datasets.