dv-load

Generates Snowflake Task DAGs and execution scripts for vault loading.

35|7|Updated Apr 14, 2022
One-click install
npx skills add https://github.com/PatrickCuba/the_data_must_flow --skill dv-load
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: dv-load
Source: https://github.com/PatrickCuba/the_data_must_flow/tree/main/dvos-skills/skills/dv-load
Command: npx skills add https://github.com/PatrickCuba/the_data_must_flow --skill dv-load

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Vault loading orchestration is complex and error-prone; this skill automates the generation of Snowflake Task DAGs and execution scripts to enforce deterministic load order and ghost record deployment.

Core Features & Use Cases

  • Generate per-hub loader DAGs with dedicated tasks and proper AFTER dependencies
  • Enforce same-hub sequential loads while parallelizing independent hubs
  • Support standard batch and Kappa Vault event-driven loading modes
  • Include ghost-record deployment for idempotent initial deployments
  • Provide reusable patterns for PIT/Bridge and satellite loading orchestration

Quick Start

Provide your vault loading requirements and run the dv-load skill to generate the Snowflake task DAGs.

Frequently Asked Questions about dv-load

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate Data Vault loads in Snowflake using task DAGs?

You can orchestrate Data Vault loads in Snowflake by generating task DAGs and execution scripts that enforce deterministic load order for hubs, links, and satellites. This automates batch vault loading and handles Kappa Vault event-driven modes.

What is the best way to enforce sequential loading for same-hub dependencies in a Snowflake ETL DAG?

To enforce sequential loading for same-hub dependencies in a Snowflake ETL DAG, you generate per-hub loader DAGs with dedicated tasks and proper AFTER dependencies. This parallelizes independent hubs while maintaining strict order for same-hub tasks.

How do I deploy idempotent ghost records for initial Data Vault deployments?

You deploy idempotent ghost records for initial Data Vault deployments by generating execution scripts that include ghost-record deployment. This ensures safe, deterministic initial loading and prevents duplicate records during vault initialization.

Can I use Snowflake tasks to orchestrate PIT and Bridge table loading alongside standard Data Vault hubs and links?

Yes, you can use Snowflake tasks to orchestrate PIT and Bridge table loading alongside standard Data Vault hubs and links. The orchestration pattern provides reusable DAG structures that automatically handle dependencies for Point-in-Time and Bridge tables when necessary.

Does standard batch Data Vault loading support Kappa Vault event-driven modes in Snowflake?

Standard batch Data Vault loading and Kappa Vault event-driven modes are both supported in Snowflake. The orchestration generates specific task DAGs and execution scripts tailored to either standard batch processing or event-driven Kappa Vault architectures.