skills.rest
jpoutrin
jpoutrinCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

sqlmesh

Generate SQLMesh model definitions for incremental data transformations.

14|1|Updated Dec 5, 2025
Search Tags:#data quality#data pipeline#data engineering#sql#data transformation#etl#sqlmesh
One-click install
npx skills add https://github.com/jpoutrin/product-forge --skill sqlmesh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sqlmesh
Source: https://github.com/jpoutrin/product-forge/tree/main/plugins/devops-data/skills/sqlmesh
Command: npx skills add https://github.com/jpoutrin/product-forge --skill sqlmesh

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Building and managing complex data transformation pipelines often lacks features like automatic DAG inference, efficient incremental processing, and isolated testing environments, leading to slower development and higher costs. This Skill provides SQLMesh patterns to address these challenges.

Core Features & Use Cases

  • Project Structure: Guidelines for organizing SQLMesh projects, including models, macros, seeds, audits, and tests.
  • Model Definition: Examples for defining various model kinds, such as FULL, INCREMENTAL_BY_TIME_RANGE, and VIEW models.
  • Virtual Environments: How to leverage virtual environments for isolated testing and development of data transformations.
  • Audits: Patterns for defining data quality audits to ensure the integrity of your data models.
  • Best Practices: Tips for using column-level lineage for impact analysis and optimizing with incremental models.
  • Use Case: Designing a new data transformation project, implementing an incremental data model for a large dataset, setting up virtual environments for data testing, or defining data quality audits for critical tables.

Quick Start

Use the sqlmesh skill to generate a SQLMesh model definition for an incremental table based on a created_at timestamp.

Frequently Asked Questions about sqlmesh

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build incremental data pipelines with automatic DAG inference?▼

SQLMesh automatically infers directed acyclic graphs (DAGs) from your model definitions, eliminating manual dependency management. Define models with incremental strategies like INCREMENTAL_BY_TIME_RANGE or INCREMENTAL_BY_UNIQUE_KEY, and SQLMesh constructs the pipeline topology and processes only changed data, reducing compute costs and runtime.

What's the best way to structure a data transformation project for ETL pipelines?▼

Organize your SQLMesh project with dedicated directories for models, macros, seeds, audits, and tests. This structure enables clear separation of concerns, simplifies maintenance, and supports virtual environments for isolated testing before production deployment.

Can I test data transformations in isolation before deploying to production?▼

Yes, SQLMesh virtual environments let you create isolated sandbox copies of your data models for testing and development. Test transformations, validate logic, and verify data quality audits without affecting production data or downstream dependencies.

How do I ensure data quality in my ETL pipelines?▼

Define data-quality audits in SQLMesh to validate model outputs against business rules and data integrity constraints. Audits catch anomalies, enforce schema expectations, and provide governance checkpoints across your pipeline, ensuring reliable downstream analytics.

Does SQLMesh support different model types for various transformation patterns?▼

SQLMesh supports multiple model kinds: FULL (complete refresh), INCREMENTAL_BY_TIME_RANGE (time-partitioned updates), INCREMENTAL_BY_UNIQUE_KEY (key-based deduplication), VIEW (logical transformations), and SEED (static reference data), accommodating diverse ETL and ELT requirements.

How can I analyze data lineage and transformation impact?▼

SQLMesh provides column-level lineage tracking to trace data flow from source to target across your pipeline. Use lineage for impact analysis, dependency discovery, and compliance audits, enabling informed decisions about schema changes and transformation modifications.

Related Skills

Explore More Data & Analytics Skills→
cameronraysmithcameronraysmith
14

preferences-data-modeling

Define data modeling conventions for database schemas and pipelines.

Community
Advanced
quangkmhdquangkmhd

data-pipeline

Automate end-to-end ETL workflows for data integration and analytics.

Community
Advanced
camilooscargbaptistacamilooscargbaptista
7

data-engineering

Assess data pipelines, ETL/ELT processes, and governance across batch and streaming systems.

Community
Advanced
minyeamerminyeamer
3

dag-factory

Convert YAML configuration files into executable Apache Airflow DAG definitions.

Community
Intermediate