skills.rest
datasurface
datasurfaceOFFICIAL
Explore all skills by this author →
GitHub

Start Initial Ingestion

Tag DataSurface Yellow model releases, provision Kubernetes secrets, and trigger Airflow ingestion DAGs.

2|Updated Jan 31, 2026
Search Tags:#automation#Kubernetes#ingestion#secrets#tagging#data-pipeline#DAG
One-click install
npx skills add https://github.com/datasurface/demo1 --skill start-initial-ingestion-datasurface
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Start Initial Ingestion
Source: https://github.com/datasurface/demo1/tree/main/.claude/skills/start-initial-ingestion
Command: npx skills add https://github.com/datasurface/demo1 --skill start-initial-ingestion-datasurface

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This guide helps users start the initial data ingestion pipeline after deploying a new DataSurface Yellow model, covering model tagging, secret provisioning, and DAG activation.

Core Features & Use Cases

  • Tag management: Validate and apply model version tags (e.g., v*..-demo) to enable ingestion workflows.
  • Secret provisioning: Create and configure required credentials to access source databases and services.
  • Ingestion DAG bootstrap: Unpause and trigger the main ingestion DAGs, and monitor their progress and logs.
  • Use Case: After a fresh model deployment, quickly bootstrap the ingestion path to ingest test data and validate end-to-end processing.

Quick Start

  1. Tag the model release (e.g., v1.0.0-demo) and push to origin.
  2. Create the required secret customer-source-credential with USER and PASSWORD.
  3. Unpause and trigger the initial ingestion DAGs (infrastructure -> factory -> ingestion) and verify logs for successful ingestion.

Frequently Asked Questions about Start Initial Ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I bootstrap a data ingestion pipeline after deploying a new model?▼

To bootstrap initial data ingestion, you must tag the model release, provision required Kubernetes secrets for source credentials, and unpause the Airflow DAGs to trigger the infrastructure, factory, and ingestion workflows.

What Kubernetes secrets do I need to configure for data ingestion?▼

You need to create a Kubernetes secret named customer-source-credential containing your USER and PASSWORD keys to authenticate the ingestion pipeline against upstream source databases and services.

How do I trigger Airflow DAGs for initial data ingestion?▼

To trigger Airflow DAGs for initial data ingestion, unpause the workflows and manually trigger the execution chain sequentially: infrastructure, then factory, and finally the ingestion DAG to process data.

Why do I need to tag a model release before starting the ingestion pipeline?▼

Tagging a model release with a version like v1.0.0-demo is required to enable ingestion workflows because it validates the deployment state and provides the reference needed to trigger downstream DAGs.

Can I trigger ingestion DAGs across remote Kubernetes environments?▼

Yes, you can trigger ingestion DAGs across local or remote environments, provided you have permissions to pause and unpause Airflow DAGs and can reference the correct Kubernetes secrets and model tags in those target clusters.

Related Skills

Explore More Data & Analytics Skills→
datasurfacedatasurface

remote-setup-walkthrough

Provision a DataSurface Yellow environment on a remote Kubernetes cluster via SSH.

Official
Advanced
RDEWAIRDEWAI
5

create-integration-test

Generates integration tests for Docker-based apps with CI/CD pipelines and automated workflows.

Official
Advanced
minyeamerminyeamer
3

linkmerce-airflow-change

Update LinkMerce Airflow DAG orchestration while validating trigger, credential, and ETL alignment.

Community
Intermediate
billynewportbillynewport

Generate DataSurface Yellow Bootstrap Artifacts

Generate Kubernetes bootstrap artifacts, Airflow DAGs, and ring initialization jobs.

Community
Advanced