ml-data-pipeline

To build a PyTorch Lightning DataModule, you can generate tailored loaders for computer vision and graph ML that handle train, validation, and test splits. This provides task-specific data loading and augmentation strategies for ready-to-train datasets.

The best way to speed up data loading is converting your datasets to LMDB. This accelerates read speeds and improves experimentation iteration by providing performance-minded data preparation for your training pipelines.

You validate dataset structure by running diagnostic scripts that check split sizes, class imbalance, image properties, and corrupted files. This deterministic preprocessing ensures your dataset is correctly formatted before training begins.

Yes, the data pipeline works with both torchvision and torch-geometric workflows. It provides task-specific loader construction and applies suitable transforms, including common vision augmentations and graph edge sampling for robustness.

You configure data augmentation strategies using Hydra-style data configuration to select suitable transforms. This applies robust augmentations like common vision techniques or graph edge sampling tailored to your specific task type.

Using LMDB for dataset acceleration requires dependencies like lmdb, pickle, and torchvision, and is optional. While it improves read speeds, you must ensure your environment supports these specific libraries to leverage the acceleration.