skills.rest
RDEWAI
RDEWAIOFFICIAL
Explore all skills by this author →
GitHub

create-integration-test

Generates integration tests for Docker-based apps with CI/CD pipelines and automated workflows.

5|1|Updated Sep 23, 2025
Search Tags:#integration testing#pytest#data quality#airflow#data pipelines#medallion architecture#unity catalog
One-click install
npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-integration-test-rdewai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: create-integration-test
Source: https://github.com/RDEWAI/Redefining-DataEngineering-With-AI/tree/main/chapter-6/developer-plugin/skills/create-integration-test
Command: npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-integration-test-rdewai

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually writing and maintaining end-to-end integration tests for medallion architecture data layers (bronze, silver, gold) that validate against a live local stack (Airflow, Unity Catalog OSS, Marquez) is tedious, error-prone, and slows down data pipeline development.

Core Features & Use Cases

  • Automated Test Suite Generation: Creates layer-scoped pytest integration test modules that trigger Airflow DAGs, validate Unity Catalog table existence and required metadata columns, and verify Spark Expectations run evidence and Marquez data quality facets.
  • Stack Readiness Checks: Automatically probes the local docker-compose stack to ensure tests skip honestly when services are unavailable, eliminating false positive test passes from unrelated processes binding ports.
  • Use Case: A data engineer building a new silver layer DAG for customer transaction processing can use this skill to auto-generate integration tests that confirm the DAG runs successfully, all expected silver tables exist in Unity Catalog, and data quality pass rates are recorded in Marquez, cutting manual test authoring time from hours to minutes.

Quick Start

Use the create-integration-test skill to generate end-to-end integration tests for the gold layer DAG associated with your approved integration-test user story.

Frequently Asked Questions about create-integration-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate integration testing for Airflow DAGs and Unity Catalog tables?▼

Automated integration testing for Airflow DAGs and Unity Catalog tables is achieved by generating layer-scoped pytest suites that trigger DAG runs, validate target table existence, and verify metadata column population without manual test authoring.

What is the best way to test data quality in a medallion architecture data pipeline?▼

Testing data quality in a medallion architecture pipeline involves generating pytest integration tests that validate Spark Expectations run evidence and verify Marquez data quality facets for bronze, silver, or gold layer transformations.

Can I generate pytest tests for Airflow DAGs that skip honestly when local services are unavailable?▼

Yes, generated pytest tests for Airflow DAGs probe the local docker-compose stack to ensure tests skip honestly when services are unavailable, eliminating false positive passes from unrelated processes binding ports.

Do I need a live local stack to validate Unity Catalog metadata and Marquez data quality facets?▼

Yes, validating Unity Catalog metadata and Marquez data quality facets requires a live local stack running Airflow, Unity Catalog OSS, and Marquez to execute the end-to-end integration tests successfully.

How to validate Airflow DAG run success and Unity Catalog table integrity without manual test authoring?▼

Validating Airflow DAG run success and Unity Catalog table integrity without manual authoring is done by auto-generating pytest integration test modules tied to integration-test user stories for bronze, silver, or gold layer transformations.

Why does my data pipeline integration test pass when Marquez or Unity Catalog OSS is not running?▼

False positive integration test passes occur when unrelated processes bind ports; generated tests probe the local docker-compose stack readiness to skip honestly when Marquez or Unity Catalog OSS services are actually unavailable.

Related Skills

Explore More Data & Analytics Skills→
RDEWAIRDEWAI
5

validate-stories

Validate sprint backlog completeness and quality before sprint planning.

Official
Advanced
i9wa4i9wa4
11

jupyter-notebook

Execute Jupyter notebooks on local Python kernels and Databricks clusters.

Community
Advanced
ivanshamaevivanshamaev
14

openlineage

Emit OpenLineage RunEvent/Job/Dataset facets to Marquez for lineage tracking.

Community
Advanced
janewilkinjanewilkin

test

Run pytest test suites and generate code coverage reports.

Community
Intermediate