skills.rest
billynewport
billynewportCOMMUNITY
Explore all skills by this author →
Website𝕏@billynewportGitHub

Start Initial Ingestion

Tag DataSurface Yellow model versions, create Kubernetes secrets, and activate Airflow ingestion DAGs.

1|Updated Feb 6, 2026
Search Tags:#kubernetes#ingestion#airflow#tagging#data-pipeline#datasurface
One-click install
npx skills add https://github.com/billynewport/demo_cokub_model --skill start-initial-ingestion
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Start Initial Ingestion
Source: https://github.com/billynewport/demo_cokub_model/tree/main/.claude/skills/start-initial-ingestion
Command: npx skills add https://github.com/billynewport/demo_cokub_model --skill start-initial-ingestion

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill guides teams to initiate the initial data ingestion pipeline after deploying a new DataSurface Yellow model, ensuring tagging, secret setup, and DAG activation are completed correctly.

Core Features & Use Cases

  • Tagging & Release Management: Ensures the model version is properly tagged for ingestion.
  • Secret & Credential Setup: Guides the creation of required Kubernetes secrets for source access.
  • DAG Activation & Orchestration: Provides steps to unpause and trigger ingestion DAGs in Airflow for a repeatable workflow.

Quick Start

  • git tag v1.0.1-demo
  • git push origin v1.0.1-demo
  • kubectl create secret generic customer-source-credential --from-literal=USER=postgres --from-literal=PASSWORD=password -n demo1
  • kubectl exec -n demo1 deployment/airflow-api-server -- airflow dags unpause scd2__CustomerDB_ingestion
  • kubectl exec -n demo1 deployment/airflow-api-server -- airflow dags trigger scd2__CustomerDB_ingestion

Frequently Asked Questions about Start Initial Ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I start the initial data ingestion pipeline after deploying a DataSurface model?▼

To start initial data ingestion, you must tag your model version in git, create the required Kubernetes secrets for source access, and then unpause and trigger the Airflow ingestion DAGs to execute the workflow.

What prerequisites are needed to trigger Airflow ingestion DAGs in Kubernetes?▼

Triggering Airflow ingestion DAGs requires a properly tagged git release, configured Kubernetes secrets containing source credentials, and ensuring the target DAGs are unpaused within the Airflow environment to guarantee a reproducible setup.

How do I create Kubernetes secrets for data pipeline source access?▼

You create Kubernetes secrets for data pipeline access using the kubectl create secret command with literal values for credentials, specifying the namespace to ensure the ingestion DAGs can authenticate with the source systems.

Why does my Airflow ingestion DAG need to be unpaused before triggering?▼

Airflow ingestion DAGs must be unpaused before triggering to allow the scheduler to process and execute tasks. Paused DAGs prevent the pipeline from running, blocking the initial data ingestion workflow and model processing.

Does data ingestion with DataSurface require a specific git tagging workflow?▼

Yes, data ingestion requires a specific git tagging workflow where you tag the model version and push it to the remote repository, ensuring an auditable and reproducible release before the Airflow DAGs are activated.

What happens if I trigger an Airflow DAG without setting up Kubernetes secrets?▼

Triggering an Airflow DAG without Kubernetes secrets causes the ingestion pipeline to fail because the workflow cannot authenticate with the source database, halting data extraction and breaking the reproducible setup.

Related Skills

Explore More Data & Analytics Skills→
billynewportbillynewport

Generate DataSurface Yellow Bootstrap Artifacts

Generate Kubernetes bootstrap artifacts, Airflow DAGs, and ring initialization jobs.

Community
Advanced
rory-datarory-data

testing-dags

Trigger Airflow DAG runs, diagnose failures, inspect logs, and apply fixes in iterative cycles.

Community
Advanced
futureteafuturetea
1

rancher-resource-discovery

Inventory Kubernetes resources and map dependencies across clusters.

Community
Advanced
borgheiborghei
467

senior-ml-engineer

Provide production-grade ML engineering guidance and templates for end-to-end projects.

Community
Advanced