Start Initial Ingestion

Tag DataSurface Yellow model releases, provision Kubernetes secrets, and trigger Airflow ingestion DAGs.

2|Updated Jan 31, 2026

Search Tags:#automation #Kubernetes #ingestion #secrets #tagging #data-pipeline #DAG

Ready to run

npx skills add https://github.com/datasurface/demo1 --skill start-initial-ingestion-datasurface

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Start Initial Ingestion
Source: https://github.com/datasurface/demo1/tree/main/.claude/skills/start-initial-ingestion
Command: npx skills add https://github.com/datasurface/demo1 --skill start-initial-ingestion-datasurface

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This guide helps users start the initial data ingestion pipeline after deploying a new DataSurface Yellow model, covering model tagging, secret provisioning, and DAG activation.

Core Features & Use Cases

Tag management: Validate and apply model version tags (e.g., v*..-demo) to enable ingestion workflows.
Secret provisioning: Create and configure required credentials to access source databases and services.
Ingestion DAG bootstrap: Unpause and trigger the main ingestion DAGs, and monitor their progress and logs.
Use Case: After a fresh model deployment, quickly bootstrap the ingestion path to ingest test data and validate end-to-end processing.

Quick Start

Tag the model release (e.g., v1.0.0-demo) and push to origin.
Create the required secret customer-source-credential with USER and PASSWORD.
Unpause and trigger the initial ingestion DAGs (infrastructure -> factory -> ingestion) and verify logs for successful ingestion.

Frequently Asked Questions about Start Initial Ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I bootstrap a data ingestion pipeline after deploying a new model?▼

To bootstrap initial data ingestion, you must tag the model release, provision required Kubernetes secrets for source credentials, and unpause the Airflow DAGs to trigger the infrastructure, factory, and ingestion workflows.

What Kubernetes secrets do I need to configure for data ingestion?▼

You need to create a Kubernetes secret named customer-source-credential containing your USER and PASSWORD keys to authenticate the ingestion pipeline against upstream source databases and services.

How do I trigger Airflow DAGs for initial data ingestion?▼

To trigger Airflow DAGs for initial data ingestion, unpause the workflows and manually trigger the execution chain sequentially: infrastructure, then factory, and finally the ingestion DAG to process data.

Why do I need to tag a model release before starting the ingestion pipeline?▼

Tagging a model release with a version like v1.0.0-demo is required to enable ingestion workflows because it validates the deployment state and provides the reference needed to trigger downstream DAGs.

Can I trigger ingestion DAGs across remote Kubernetes environments?▼

Yes, you can trigger ingestion DAGs across local or remote environments, provided you have permissions to pause and unpause Airflow DAGs and can reference the correct Kubernetes secrets and model tags in those target clusters.