skills.rest
mozilla
mozillaOFFICIAL
Explore all skills by this author →
WebsiteGitHub

metadata-manager

Create BigQuery ETL metadata and schema YAML files with descriptions.

10|2|Updated Nov 6, 2025
Search Tags:#ETL#BigQuery#Automation#Schema#Airflow DAG#Metadata#Data Governance
One-click install
npx skills add https://github.com/mozilla/bigquery-etl-skills --skill metadata-manager
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: metadata-manager
Source: https://github.com/mozilla/bigquery-etl-skills/tree/main/metadata-manager
Command: npx skills add https://github.com/mozilla/bigquery-etl-skills --skill metadata-manager

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires PyYAML, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

Managing BigQuery table metadata, schemas, and Airflow DAG configurations manually is complex, time-consuming, and prone to inconsistencies. This skill streamlines the entire process, ensuring data governance and operational efficiency.

Core Features & Use Cases

  • Intelligent Schema & Metadata Generation: Automatically creates schema.yaml and metadata.yaml files with intelligent descriptions, leveraging various data sources (local files, Glean Dictionary, DataHub).
  • DAG Configuration Management: Helps find existing Airflow DAGs or creates new ones, inferring optimal scheduling, ownership, and impact tiers.
  • Test Coordination: Coordinates updates to unit tests (invoking sql-test-generator) when BigQuery queries are modified, ensuring tests remain in sync with query logic.
  • Description Improvement: Proactively detects missing or vague descriptions in source tables and offers to improve them, enhancing metadata completeness across the repository.
  • Use Case: A data team needs to create a new derived BigQuery table. Instead of manually drafting YAML files for schema, metadata, and Airflow DAGs, they use this skill to generate all configurations, auto-populating descriptions and ensuring compliance with bigquery-etl standards.

Quick Start

Generate schema.yaml and metadata.yaml for the new BigQuery table my-project.my-dataset.new_table_v1. Infer the best Airflow DAG based on existing patterns.

Frequently Asked Questions about metadata-manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate BigQuery metadata and schema generation for ETL pipelines?▼

Automate BigQuery metadata by using this skill to generate schema.yaml and metadata.yaml files with intelligent descriptions from multiple sources (local SQL files, Glean Dictionary, DataHub), eliminating manual YAML creation and ensuring consistent data governance across your ETL pipelines.

Can I automatically generate Airflow DAG configurations from existing patterns?▼

Yes. The skill infers optimal Airflow DAG configurations by analyzing existing patterns in your repository, automatically determining scheduling, ownership, and impact tiers without manual YAML drafting.

How do I keep BigQuery tests in sync when queries change?▼

The skill coordinates test updates by invoking sql-test-generator whenever BigQuery queries are modified, ensuring unit tests remain synchronized with query logic and reducing manual test maintenance.

What's the best way to detect and improve missing table descriptions?▼

The skill proactively scans source tables for missing or incomplete descriptions and offers automated improvements, sourcing descriptions from SQL comments, Glean Dictionary, or DataHub to enhance metadata completeness.

Does this work with existing bigquery-etl repository standards?▼

Yes. The skill generates schema, metadata, and DAG configurations compliant with bigquery-etl standards, automatically applying repository conventions and integrating with core skills and sql-test-generator.

Can I use this skill to create new BigQuery tables with complete configurations?▼

Yes. Provide a new BigQuery table identifier and the skill generates all required configurations—schema.yaml, metadata.yaml, and inferred Airflow DAG—with auto-populated descriptions in a single operation.

Related Skills

Explore More Data & Analytics Skills→
mozillamozilla
10

model-requirements

Gather BigQuery data model requirements and validate dependencies for bqetl queries.

Official
Advanced
RDEWAIRDEWAI
5

create-integration-test

Generates integration tests for Docker-based apps with CI/CD pipelines and automated workflows.

Official
Advanced
k1lgork1lgor
2

workflow-orchestrator

Orchestrate scheduled and event-driven workflows across Airflow, Celery, and Cron.

Community
Advanced
astronomerastronomer
419

annotating-task-lineage

Annotate Airflow task lineage with inlets and outlets using OpenLineage datasets.

Official
Intermediate