Start Initial Ingestion

Tag DataSurface Yellow model versions, create Kubernetes secrets, and activate Airflow ingestion DAGs.

1|Updated Feb 6, 2026

Search Tags:#kubernetes #ingestion #airflow #tagging #data-pipeline #datasurface

Ready to run

npx skills add https://github.com/billynewport/demo_cokub_model --skill start-initial-ingestion

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Start Initial Ingestion
Source: https://github.com/billynewport/demo_cokub_model/tree/main/.claude/skills/start-initial-ingestion
Command: npx skills add https://github.com/billynewport/demo_cokub_model --skill start-initial-ingestion

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill guides teams to initiate the initial data ingestion pipeline after deploying a new DataSurface Yellow model, ensuring tagging, secret setup, and DAG activation are completed correctly.

Core Features & Use Cases

Tagging & Release Management: Ensures the model version is properly tagged for ingestion.
Secret & Credential Setup: Guides the creation of required Kubernetes secrets for source access.
DAG Activation & Orchestration: Provides steps to unpause and trigger ingestion DAGs in Airflow for a repeatable workflow.

Quick Start

git tag v1.0.1-demo
git push origin v1.0.1-demo
kubectl create secret generic customer-source-credential --from-literal=USER=postgres --from-literal=PASSWORD=password -n demo1
kubectl exec -n demo1 deployment/airflow-api-server -- airflow dags unpause scd2__CustomerDB_ingestion
kubectl exec -n demo1 deployment/airflow-api-server -- airflow dags trigger scd2__CustomerDB_ingestion

Frequently Asked Questions about Start Initial Ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I start the initial data ingestion pipeline after deploying a DataSurface model?▼

To start initial data ingestion, you must tag your model version in git, create the required Kubernetes secrets for source access, and then unpause and trigger the Airflow ingestion DAGs to execute the workflow.

What prerequisites are needed to trigger Airflow ingestion DAGs in Kubernetes?▼

Triggering Airflow ingestion DAGs requires a properly tagged git release, configured Kubernetes secrets containing source credentials, and ensuring the target DAGs are unpaused within the Airflow environment to guarantee a reproducible setup.

How do I create Kubernetes secrets for data pipeline source access?▼

You create Kubernetes secrets for data pipeline access using the kubectl create secret command with literal values for credentials, specifying the namespace to ensure the ingestion DAGs can authenticate with the source systems.

Why does my Airflow ingestion DAG need to be unpaused before triggering?▼

Airflow ingestion DAGs must be unpaused before triggering to allow the scheduler to process and execute tasks. Paused DAGs prevent the pipeline from running, blocking the initial data ingestion workflow and model processing.

Does data ingestion with DataSurface require a specific git tagging workflow?▼

Yes, data ingestion requires a specific git tagging workflow where you tag the model version and push it to the remote repository, ensuring an auditable and reproducible release before the Airflow DAGs are activated.

What happens if I trigger an Airflow DAG without setting up Kubernetes secrets?▼

Triggering an Airflow DAG without Kubernetes secrets causes the ingestion pipeline to fail because the workflow cannot authenticate with the source database, halting data extraction and breaking the reproducible setup.