Design Airbyte ELT pipelines with sync modes, cursors, and dbt normalization.
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airbyte
Please help me install this Agent Skill. Skill: airbyte Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/airbyte Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airbyte
Airbyte ELT helps teams reliably move data from sources to destinations while supporting full refresh and incremental syncing, including deduped incremental upserts and schema evolution handling.
_airbyte_raw_ / meta-enriched structures and optionally perform dbt-based basic normalization for typed tables.Use the airbyte skill to design an Airbyte pipeline that syncs a PostgreSQL “orders” source into Snowflake using Incremental Append + Deduped with updated_at as the cursor and order_id as the primary key.
High-intent search queries and answers about installing and using this skill.
To configure an Airbyte incremental sync, select Incremental Append + Deduped mode, specify cursor fields like `updated_at` and primary keys like `order_id` for per-stream policies to capture changes and deduplicate records in the analytical destination.
Airbyte full refresh overwrites or appends entire snapshot data per sync, whereas incremental append captures only new records using stateful cursors, and incremental append + deduped merges them using primary keys to maintain a current dataset view.
Yes, you can manage Airbyte connections via the REST API or Terraform for infrastructure-as-code automation, and orchestrate the resulting ELt sync schedules directly from Apache Airflow for production pipeline execution.
Airbyte handles schema evolution using non-breaking schema update policies to automatically adapt to new columns, while clearly defining breaking change behaviors to prevent pipeline failures when source structures unexpectedly shift.
Airbyte ELT pipelines can be deployed in production using abctl for local management, Helm for Kubernetes environments, or Airbyte Cloud, enabling sync health monitoring and alerting readiness through UI logs and metrics.
Yes, to use Incremental Append + Deduped sync modes, you must specify cursor fields for change capture and primary keys for deduplication on a per-stream basis within the Airbyte connection catalog metadata.
underPressueOrchestrate, validate, and optimize ETL/ELT data pipelines in Python.
logicalclocksQuery Hopsworks feature store tables using Trino SQL.
manutejAutomate Apache Airflow DAG definitions, task dependencies, and scheduling.
nrakowSelect and configure data ingestion strategies for Fivetran, Airbyte, or custom pipelines.