skills.rest
RDEWAI
RDEWAIOFFICIAL
Explore all skills by this author →
GitHub

generate-se-rules

Convert DQS markdown documents into per-table Spark-Expectations YAML rule files.

5|1|Updated Sep 23, 2025
Search Tags:#yaml#generate#data-quality#data-engineering#dqs#spark-expectations#rule-conversion
One-click install
npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill generate-se-rules
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: generate-se-rules
Source: https://github.com/RDEWAI/Redefining-DataEngineering-With-AI/tree/main/chapter-6/dq-engineer-plugin/skills/generate-se-rules
Command: npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill generate-se-rules

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sqlparse, pyyaml, and includes scripts (resource) components.

What problem does it solve?

Manually converting Data Quality Specification (DQS) markdown documents into per-table Spark-Expectations YAML rule files is time-consuming, error-prone, and difficult to maintain for large data pipelines with hundreds of quality rules.

Core Features & Use Cases

  • Automated DQS Parsing: Extracts field-level, referential integrity, statistical, and reconciliation rules from structured DQS markdown files.
  • Per-Table YAML Generation: Groups rules by target table and produces one valid, spark-expectations >=2.6.0 compatible YAML file per table with correct environment-specific configurations.
  • Built-in Validation & Learning Capture: Validates generated YAML against SE schema constraints, handles edge cases like SCD2 metadata columns and ambiguous table references, and automatically records user corrections to improve future rule generation.
  • Use Case: A data quality engineer working on a patient data pipeline can use this skill to generate all required Spark-Expectations rule configs from a single DQS document in minutes, instead of writing and debugging hundreds of YAML entries manually.

Quick Start

Provide the path to your DQS markdown file to the generate-se-rules skill to automatically produce per-table Spark-Expectations YAML rule files saved to the outputs/dqs directory.

Frequently Asked Questions about generate-se-rules

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert DQS markdown documents into Spark-Expectations YAML rules?▼

To convert Data Quality Specification markdown documents into Spark-Expectations YAML rules, provide the DQS file path to the parser. It automatically extracts field-level and referential rules, grouping them by target table to output valid per-table YAML rule files.

What is the best way to automate data quality rule generation for Spark pipelines?▼

Automating data quality rule generation for Spark pipelines is best handled by parsing centralized DQS documentation into per-table Spark-Expectations YAML configs. This approach eliminates manual YAML writing, ensuring automated data quality enforcement across large pipelines.

Does Spark-Expectations YAML rule generation support environment-specific configurations?▼

Yes, Spark-Expectations YAML rule generation supports environment-specific configurations. The conversion process maps DQS rule types to correct SE rule types, producing valid spark-expectations >=2.6.0 compatible rule files tailored to your required environments.

How do I validate generated YAML rules against Spark-Expectations schema constraints?▼

To validate generated YAML rules against Spark-Expectations schema constraints, the rule generation process includes built-in inline validation. It automatically checks syntax, handles edge cases like SCD2 metadata columns, and records user corrections for future improvements.

Can I generate per-table YAML files from a single Data Quality Specification document?▼

Yes, you can generate per-table YAML files from a single Data Quality Specification document. The parser automatically extracts all rules, groups them by target table, and outputs one valid YAML file per table to your specified outputs directory.

What are the limitations of parsing DQS markdown for Spark-Expectations rule conversion?▼

Limitations of parsing DQS markdown for Spark-Expectations rule conversion include handling ambiguous table references and SCD2 metadata columns, which require careful edge case validation. Additionally, the generated YAML strictly requires spark-expectations >=2.6.0 compatibility to function correctly.

Related Skills

Explore More Data & Analytics Skills→
RDEWAIRDEWAI
5

validate-dqs

Validate Data Quality Specification documents for completeness and quality standards.

Official
Advanced
vivekganavivekgana
4

data-quality

Automate data quality validation in Spark/Databricks pipelines with Great Expectations and DLT checks.

Community
Advanced
RDEWAIRDEWAI
5

validate-ingestion

Validate Bronze ingestion artifacts against LLD specifications and report issues.

Official
Advanced
RDEWAIRDEWAI
5

create-dqs

Generate build-ready Data Quality Specifications from approved STM, DMS, and DRD artifacts for Spark-Expectations integration.

Official
Advanced