ingest

Ingest data from diverse sources into a data lake or warehouse.

209|30|Updated Sep 18, 2021

Search Tags:#etl #data-pipeline #batch-processing #ingest #data-ingestion #starlake

Ready to run

npx skills add https://github.com/starlake-ai/starlake --skill ingest

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ingest
Source: https://github.com/starlake-ai/starlake/tree/main/.agent/skills/ingest
Command: npx skills add https://github.com/starlake-ai/starlake --skill ingest

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Ingest data from diverse sources into your data lake or data warehouse, simplifying loading for analytics and reporting.

Core Features & Use Cases

Ingest from multiple sources with domain, schema, and paths configuration.
Schedule and parameterize loads to support repeatable pipelines for analytics, migrations, and incremental updates.
Use cases include batch data loads, data migration, and incremental ingestion into your warehouse.

Quick Start

Ingest data from a specified path into the target domain and schema with a single command.

Frequently Asked Questions about ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I ingest data from multiple sources into a data warehouse?▼

Batch data ingestion from diverse sources into a data warehouse or data lake is configured using domain, schema, and paths settings to simplify loading for analytics and reporting pipelines.

Can I schedule repeatable batch data loads for incremental updates?▼

Yes, batch ingestion workflows support load scheduling and parameterization, enabling repeatable data pipelines for incremental ingestion, reporting, and data migrations into your warehouse.

What is the best way to load data into a data lake for analytics pipelines?▼

Loading data into a data lake for analytics is handled through batch ingestion workflows that satisfy domain, schema, paths, and scheduling configuration requirements in a single command.

Does this data ingestion method support data migration workflows?▼

Data migration is a supported use case, utilizing parameterized loads and batch ingestion to effectively transfer data from specified paths into target domains and schemas.

Do I need to configure schema and domain settings for batch data ingestion?▼

Yes, specifying the target domain, schema, and paths is required to define how diverse source data is structured and loaded into your data lake or warehouse during batch processing.