skills.rest
HermeticOrmus
HermeticOrmusCOMMUNITY
Explore all skills by this author →
Website𝕏@dbodartmGitHub

data-versioning-patterns

Implement reproducible ML data management with DVC, Delta Lake, and OpenLineage.

Updated Feb 25, 2026
Search Tags:#mlops#data lineage#reproducibility#dvc#openlineage#data versioning#delta lake
One-click install
npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill data-versioning-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-versioning-patterns
Source: https://github.com/HermeticOrmus/LibreMLOps-Claude-Code/tree/main/plugins/data-versioning/skills/data-versioning-patterns
Command: npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill data-versioning-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the critical challenge of managing and versioning datasets in machine learning projects, ensuring reproducibility and auditability.

Core Features & Use Cases

  • Version Control for Data: Integrates with tools like DVC to track datasets alongside code.
  • Reproducible Pipelines: Defines ML workflows in dvc.yaml for end-to-end reproducibility.
  • Experiment Management: Facilitates hyperparameter tuning and experiment tracking without cluttering Git history.
  • Time Travel for Datasets: Leverages Delta Lake for accessing historical data versions.
  • Lineage Tracking: Supports OpenLineage for emitting metadata about data processing jobs.
  • Use Case: Ensure that a specific model version can be retrained with the exact same data it was originally trained on, or audit the data used for a particular prediction.

Quick Start

Initialize DVC in your Git repository and track a dataset named 'raw_data.csv' by running dvc init followed by dvc add raw_data.csv.

Frequently Asked Questions about data-versioning-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I manage data versioning for machine learning datasets alongside code?▼

Data versioning for machine learning datasets integrates DVC with Git to track large files. This ensures that specific model versions can always be retrained using their exact original training data.

What is the best way to ensure reproducible ML pipelines?▼

Defining ML workflows in a dvc.yaml file ensures reproducible ML pipelines end-to-end. This approach allows you to cache intermediate results and automatically regenerate outputs when dependencies or hyperparameters change.

Can I use Delta Lake for time travel on historical dataset versions?▼

Yes, Delta Lake provides time travel capabilities for accessing historical data versions. This allows you to query and audit previous states of your datasets for debugging and compliance.

How does OpenLineage support data lineage tracking?▼

OpenLineage supports data lineage tracking by emitting metadata about data processing jobs. This creates an auditable record of how data moves and transforms across your workflows.

Do I need Git to run DVC experiments for hyperparameter search?▼

Yes, DVC experiments integrate directly with your existing Git repository. They facilitate hyperparameter tuning and experiment tracking by capturing variations without cluttering your main Git history with numerous commits.

Related Skills

Explore More Software Engineering Skills→
Alanlee0323Alanlee0323

using-dvc

Version data and ML artifacts with DVC to keep git lightweight.

Community
Intermediate
ivanshamaevivanshamaev
14

openlineage

Emit OpenLineage RunEvent/Job/Dataset facets to Marquez for lineage tracking.

Community
Advanced
beauschwabbeauschwab

annotating-task-lineage

Annotate Airflow tasks with manual lineage metadata using inlets and outlets.

Community
Intermediate
OpenCovenOpenCoven
20

mlflow

Track experiments, version models, and deploy MLflow models.

Official
Advanced