ml-data-pipeline

What problem does it solve?

Creating reliable data loading, preprocessing, and augmentation pipelines is time-consuming and error-prone, especially when you need to match your task type (vision, NLP, graph ML) and keep training throughput high.

Core Features & Use Cases

• Task-aware DataModule generation: Produce PyTorch Lightning DataModules tailored to computer vision and graph ML patterns (train/val/test setup and loaders).
• Augmentation and preprocessing guidance: Apply suitable transforms for robustness (e.g., common vision augmentations; graph transforms like edge sampling).
• Performance-minded data preparation: Convert datasets to LMDB for faster reads and easier experimentation iteration.
• Validation and dataset diagnostics: Run dataset checks for structure, split sizes, class imbalance, image properties, and corrupted files.

Quick Start

Use the ml-data-pipeline skill to generate a PyTorch Lightning DataModule and scripts for preprocessing, validating, and (optionally) accelerating your dataset with LMDB.