skills.rest
RDEWAI
RDEWAIOFFICIAL
Explore all skills by this author →
GitHub

create-dqs

Generate build-ready Data Quality Specifications from approved STM, DMS, and DRD artifacts for Spark-Expectations integration.

5|1|Updated Sep 23, 2025
Search Tags:#data quality#data governance#validation rules#medallion pipelines#dqs generation#spark-expectations
One-click install
npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-dqs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: create-dqs
Source: https://github.com/RDEWAI/Redefining-DataEngineering-With-AI/tree/main/chapter-6/dq-engineer-plugin/skills/create-dqs
Command: npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-dqs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openpyxl.

What problem does it solve?

Manually creating data quality specifications is time-consuming, error-prone, and often results in incomplete rules that miss upstream defects or lack traceability to upstream engineering artifacts. This Skill eliminates that manual work by enforcing a structured, gate-driven workflow that produces complete, compliant, and build-ready DQS documents aligned to enterprise data quality standards.

Core Features & Use Cases

  • Structured Elicitation Workflow: Uses targeted Q&A to gather all required DQ design decisions (severity thresholds, alert channels, tolerance levels) before generating output, eliminating vague or missing rules.
  • Mandatory Upstream Validation Gates: Enforces approval of all upstream STM, DMS, and DRD artifacts and validates source data volumes via read-only database queries to ensure statistical baselines are data-driven.
  • Spark-Expectations Integration: Auto-generates compatible SE YAML rule files after DQS validation, with correct rule typing and field semantics for seamless pipeline deployment.
  • Use Case: A data quality engineer working on a healthcare medallion pipeline can use this Skill to translate approved STM mappings, DMS schemas, and DRD requirements into a full DQS covering field validations, referential integrity, statistical tests, reconciliation, and SLA monitoring in minutes.

Quick Start

Use the create-dqs skill to generate a complete, compliant data quality specification for your latest approved STM, DMS, and DRD artifacts, including all validation rules, reconciliation checks, and alert frameworks for your medallion data pipeline.

Frequently Asked Questions about create-dqs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate data quality specification creation for medallion data pipelines?▼

Automating data quality specification creation involves translating approved Source-to-Target Mapping, Data Model Specification, and Data Requirements Document artifacts into a build-ready DQS. This enforces field-level validation rules, referential integrity, and SLA monitoring aligned to enterprise standards.

What is a Data Quality Specification in the context of data governance?▼

A Data Quality Specification in data governance is a traceable, auditable document defining field-level validation rules, referential integrity checks, and statistical distribution tests. It ensures medallion pipeline data meets enterprise DQ standards before deployment.

Can I generate Spark-Expectations YAML rule files from a source-to-target mapping?▼

Yes, you can generate Spark-Expectations YAML rule files from a source-to-target mapping. The workflow validates the mapping, applies data-driven statistical baselines, and auto-generates compatible YAML files with correct rule typing for pipeline deployment.

Do I need approved upstream artifacts before generating data validation rules?▼

Yes, you need approved upstream artifacts before generating data validation rules. The workflow enforces mandatory validation gates for Source-to-Target Mapping, Data Model Specification, and Data Requirements Documents to ensure complete, compliant DQS generation.

What's the best way to ensure data quality rules are traceable to upstream engineering artifacts?▼

The best way to ensure data quality rules are traceable to upstream engineering artifacts is using a gate-driven workflow that validates Source-to-Target Mapping and Data Model Specifications before generating a build-ready Data Quality Specification document.

How does statistical baseline validation work for data quality engineering?▼

Statistical baseline validation for data quality engineering works by executing read-only database queries against source data volumes. This ensures statistical distribution tests and tolerance levels in the Data Quality Specification are data-driven and accurate.

Related Skills

Explore More Data & Analytics Skills→
RDEWAIRDEWAI
5

create-dms

Generate compliant Data Model Specification documents from HLD and DRD inputs.

Official
Advanced
RDEWAIRDEWAI
5

generate-se-rules

Convert DQS markdown documents into per-table Spark-Expectations YAML rule files.

Official
Advanced
RDEWAIRDEWAI
5

validate-dqs

Validate Data Quality Specification documents for completeness and quality standards.

Official
Advanced
mgher668mgher668

design-html

Convert approved mockups and planning documents into production HTML/CSS.

Community
Advanced