lustre-safe-compress

What problem does it solve?

On Lustre filesystems near quota limits, naive bulk compression can silently corrupt or delete data when writes fail mid-operation. This Skill provides a per-file safety pipeline that compresses files with verification and multi-pass retries to prevent data loss and ensure recoverable progress across quota races.

Core Features & Use Cases

• Per-file safe compression: compress each file with a per-file verify step before deletion of the original.
• OST quota race handling: multi-pass retry pattern to converge on a zero-loss compression workflow under quota pressure.
• HPC-friendly workflow: designed for integration with batch scheduling (e.g., SLURM) and large-scale datasets (e.g., tickers, numpy arrays).

Quick Start

Run the Lustre-safe-compress workflow on your target data directory to safely compress files with per-file verification and multi-pass retries.