creating-openlineage-extractors

Create custom OpenLineage extractors for Airflow operators lacking built-in lineage support.

2|Updated Apr 8, 2026
One-click install
npx skills add https://github.com/miptah21/skills --skill creating-openlineage-extractors-miptah21
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: creating-openlineage-extractors
Source: https://github.com/miptah21/skills/tree/main/.agents/skills/creating-openlineage-extractors
Command: npx skills add https://github.com/miptah21/skills --skill creating-openlineage-extractors-miptah21

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you restore OpenLineage lineage when Airflow tasks come from operators that do not provide built-in OpenLineage support, especially when you need richer results than inlets/outlets can express.

Core Features & Use Cases

  • Build custom OpenLineage extractors to emit input/output dataset lineage for third-party or provider operators you cannot modify.
  • Add runtime and failure-aware lineage using extractor hooks so lineage can reflect what actually happened during task execution.
  • Optionally enrich outputs with facets to support advanced needs like statistics and (if desired) column-level lineage via OpenLineage facets.
  • Operational guidance and guardrails on approach selection, registration, and safe extraction patterns to prevent lineage from breaking tasks.

Quick Start

Register your extractor class in Airflow by setting AIRFLOW__OPENLINEAGE__EXTRACTORS to the import path for your custom extractor so Airflow begins emitting lineage events for the targeted operator.

Frequently Asked Questions about creating-openlineage-extractors

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I capture OpenLineage lineage for Airflow operators that lack built-in support?

To capture OpenLineage lineage for unsupported Airflow operators, you create custom extractors or operator lineage methods that map operator properties to OpenLineage datasets, then register them via the AIRFLOW__OPENLINEAGE__EXTRACTORS configuration.

When do I need a custom OpenLineage extractor instead of using Airflow inlets and outlets?

You need a custom OpenLineage extractor when third-party operators lack built-in support, when inputs and outputs are determined at runtime, or when you need richer lineage details like failure-aware metadata that inlets and outlets cannot express.

How do I register a custom OpenLineage extractor in Airflow?

You register a custom OpenLineage extractor in Airflow by setting the AIRFLOW__OPENLINEAGE__EXTRACTORS configuration to the import path of your custom extractor class, enabling Airflow to emit lineage events for the targeted operator.

Can I add column-level lineage and output statistics using OpenLineage extractors?

Yes, you can optionally enrich OpenLineage extractors with facets to support advanced needs like output statistics and column-level lineage, providing deeper metadata about the datasets involved in your Airflow tasks.

How do I safely implement OpenLineage extraction without breaking Airflow task execution?

You safely implement OpenLineage extraction by following operational guardrails for approach selection and registration, ensuring that custom extractor hooks handle lineage capture without disrupting the actual Airflow task execution.

Does OpenLineage work with third-party provider operators in Airflow?

OpenLineage works with third-party provider operators by implementing custom extractors that define operator hooks, allowing you to emit input and output dataset lineage for operators you cannot directly modify.