skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

datahub-catalog

Automate DataHub metadata ingestion and lineage capture via YAML recipes and Python SDK.

14|1|Updated May 5, 2026
Search Tags:#lineage#datahub#kubernetes helm#metadata ingestion#finegrainedlineage#dbt artifacts#graphql search
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill datahub-catalog
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: datahub-catalog
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/datahub_catalog
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill datahub-catalog

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill solves the problem of setting up DataHub metadata ingestion so you can centralize datasets, capture lineage, and enable search/discovery for data engineering assets.

Core Features & Use Cases

  • Deploy and operate DataHub: Use Docker Compose quickstart or Kubernetes Helm deployment, including Kafka/Elasticsearch/MySQL components.
  • Ingest metadata from many sources: Generate and run ingestion recipes for PostgreSQL, Hive, Spark, dbt, Airflow, Kafka, and S3, including stateful ingestion and optional profiling.
  • Emit metadata programmatically with Python SDK: Use DatahubRestEmitter and MCP wrappers to publish entity aspects like schema, ownership, tags, glossary terms, and lineage.
  • Support lineage at dataset and column granularity: Attach table-level upstream lineage and FineGrainedLineage field-to-field mappings.
  • Discover assets: Use DataHub UI, REST search APIs, or GraphQL lineage traversal to navigate impact and dependencies.

Quick Start

Activate this skill and ask your agent to ingest metadata from a dbt project by preparing a dbt ingestion recipe and running datahub ingest against your DataHub GMS endpoint.

Frequently Asked Questions about datahub-catalog

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest metadata from dbt artifacts into DataHub?▼

DataHub metadata ingestion centralizes datasets by generating and running YAML ingestion recipes against the DataHub GMS endpoint. This captures schemas and lineage from sources like dbt, PostgreSQL, and Spark.

Can I capture fine-grained lineage at the column level in DataHub?▼

Yes, DataHub supports fine-grained lineage to capture column-level dependencies. You can attach table-level upstream lineage and field-to-field mappings using the Python REST SDK and MetadataChangeProposalWrapper.

How do I deploy DataHub on Kubernetes?▼

You can deploy DataHub on Kubernetes using Helm charts. This deployment involves configuring Kafka, Elasticsearch, and MySQL components to operate the metadata catalog infrastructure.

Does DataHub support programmatic metadata emission via Python?▼

Yes, DataHub supports programmatic metadata emission through the Python REST SDK. You use DatahubRestEmitter to publish entity aspects like schema, ownership, tags, glossary terms, and lineage.

What is the best way to search and traverse lineage in DataHub?▼

The best way to search and traverse DataHub lineage is using GraphQL APIs or REST search endpoints. This allows you to navigate dataset dependencies and assess impact across your data engineering assets.

Related Skills

Explore More Data & Analytics Skills→
therealbilltherealbill
3

cluster-deployment

Automate self-hosted Temporal cluster deployment on Kubernetes using Helm.

Community
Advanced
SciMate-AISciMate-AI
1

reactome-database

Query Reactome REST APIs to retrieve pathway data and perform enrichment analyses.

Official
Intermediate
ouakarouakar

data-engineer

Automate end-to-end data pipelines from ingestion to analytics-ready datasets.

Community
Advanced
Pear-CommercePear-Commerce

vendor-recipe-ingestor

Parse vendor recipe sites lacking LD+JSON markup with custom DOM parsers.

Official
Advanced