lineage

Generate task dependency graphs for Starlake pipelines with DOT, SVG, JSON, or PNG output.

209|30|Updated Sep 18, 2021
One-click install
npx skills add https://github.com/starlake-ai/starlake --skill lineage
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: lineage
Source: https://github.com/starlake-ai/starlake/tree/main/.agent/skills/lineage
Command: npx skills add https://github.com/starlake-ai/starlake --skill lineage

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Graphing data lineage and task dependencies to understand how data and tasks flow across a Starlake pipeline, enabling governance and debugging.

Core Features & Use Cases

  • Generate a graph of upstream and downstream task relationships to reveal data lineage.
  • Export representations in DOT, SVG, JSON, or PNG for reporting and visualization.
  • Use Case: Identify which tasks contribute to a given downstream artifact and spot circular dependencies or orphan tasks.

Quick Start

Run the lineage command to generate a task-dependency graph for your project and export it to lineage.dot.

Frequently Asked Questions about lineage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I visualize ETL task dependencies and data lineage?

To visualize ETL task dependencies and data lineage, generate a graph of upstream and downstream task relationships across your pipelines. This reveals how data flows and surfaces task bottlenecks within your project.

Can I export data lineage graphs to SVG, JSON, or PNG formats?

Yes, you can export data lineage graphs to SVG, JSON, PNG, and DOT formats. These output options support flexible reporting and visualization of your task dependency relationships.

What is data lineage graphing used for in pipeline debugging?

Data lineage graphing is used for pipeline debugging to identify which tasks contribute to a downstream artifact. It helps spot circular dependencies, orphan tasks, and upstream/downstream bottlenecks.

How do I identify orphan tasks and circular dependencies in my pipeline?

You identify orphan tasks and circular dependencies by generating a task-dependency graph for your project. This graph maps upstream and downstream relationships to reveal structural issues in your pipeline.

Does the lineage visualization support task filtering and domain reloads?

Yes, lineage visualization supports task filtering and domain reload capabilities. This allows you to focus the dependency graph on specific tasks and refresh domains as your project structure evolves.

What is the best way to analyze dataset lineage within a Starlake project?

The best way to analyze dataset lineage within a Starlake project is to run the lineage command. It generates a task-dependency graph, exportable to DOT, providing visibility into data flow and task relationships.