Emit OpenLineage RunEvent/Job/Dataset facets to Marquez for lineage tracking.
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill openlineage
Please help me install this Agent Skill. Skill: openlineage Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/openlineage Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill openlineage
This Skill helps you capture and visualize end-to-end data lineage so you can understand how datasets are produced, how columns flow through transformations, and what would be impacted by changing upstream data.
Run this skill for a Spark/airflow pipeline by installing and configuring the OpenLineage integration to send lineage events to a Marquez backend, then query the lineage graph for the affected dataset.
High-intent search queries and answers about installing and using this skill.
Data lineage tracking across Airflow, Spark, and dbt pipelines is done by configuring OpenLineage integrations to emit RunEvent, Job, and Dataset relationships with proper START and terminal state semantics to a backend for persistence.
Column-level lineage tracks how individual columns flow through transformations by attaching schema facets and columnLineage mappings to datasets, providing fine-grained impact analysis to understand exactly which downstream fields are affected by upstream data changes.
To set up a Marquez backend to query lineage graphs, run a local or reference backend instance, configure your OpenLineage transport to persist events there, and query namespaces, jobs, datasets, and lineage graphs via the API for impact analysis.
Yes, you can build a custom OpenLineage emitter by creating a bespoke client in Python to manually emit RunEvent, Job, and Dataset facets, ensuring lineage is captured when your tooling lacks automatic instrumentation support.
OpenLineage event modeling requires namespace consistency to reliably correlate RunEvent, Job, and Dataset facets across orchestrator and compute layers, ensuring lineage graphs persist and query correctly without disconnected or missing pipeline execution nodes.
Patasse97Design and orchestrate data pipelines with Python, SQL, Spark, Airflow, and dbt.
AltimateAIIdentify downstream consumers and column-level lineage for changed dbt models.
s1366560Design and implement scalable data pipelines with Apache Spark, dbt, and Airflow.
astronomerImplement OpenLineage extractors for Airflow operators lacking built-in lineage.
