skills.rest
KangOxford
KangOxfordCOMMUNITY
Explore all skills by this author →
GitHub

lustre-safe-compress

Compress Lustre files with per-file verification and multi-pass quota retries.

1|Updated Apr 23, 2026
Search Tags:#verification#compression#quota#hpc#zstd#lustre#data-safety
One-click install
npx skills add https://github.com/KangOxford/auto-quant-research --skill lustre-safe-compress
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: lustre-safe-compress
Source: https://github.com/KangOxford/auto-quant-research/tree/main/execution-layer/skills/lustre-safe-compress
Command: npx skills add https://github.com/KangOxford/auto-quant-research --skill lustre-safe-compress

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

On Lustre filesystems near quota limits, naive bulk compression can silently corrupt or delete data when writes fail mid-operation. This Skill provides a per-file safety pipeline that compresses files with verification and multi-pass retries to prevent data loss and ensure recoverable progress across quota races.

Core Features & Use Cases

  • Per-file safe compression: compress each file with a per-file verify step before deletion of the original.
  • OST quota race handling: multi-pass retry pattern to converge on a zero-loss compression workflow under quota pressure.
  • HPC-friendly workflow: designed for integration with batch scheduling (e.g., SLURM) and large-scale datasets (e.g., tickers, numpy arrays).

Quick Start

Run the Lustre-safe-compress workflow on your target data directory to safely compress files with per-file verification and multi-pass retries.

Frequently Asked Questions about lustre-safe-compress

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I safely compress files on a Lustre filesystem without risking data loss?▼

Safely compress Lustre files by applying a per-file pipeline that verifies integrity with zstd before deleting the original, ensuring recoverable progress without silent corruption.

Why does naive bulk compression fail on HPC storage near quota limits?▼

Naive bulk compression on HPC storage fails near quota limits because write failures mid-operation can silently corrupt or delete data when OST quota races occur during the transfer.

Does Lustre safe compression work with large numpy datasets and SLURM batch jobs?▼

Yes, this HPC-friendly compression workflow is designed for integration with SLURM batch scheduling and safely processes large-scale datasets including numpy arrays and archive directories.

How does zstd verification prevent data corruption during archive compression?▼

Zstd verification prevents data corruption by running a per-file test step to verify integrity, and only proceeding with guarded deletion of the original file after successful verification.

What is the best way to handle OST quota races when freeing space on Lustre?▼

Handle OST quota races on Lustre using a multi-pass retry pattern that converges on a zero-loss compression workflow, ensuring idempotent correctness even under heavy quota pressure.

Related Skills

Explore More Software Engineering Skills→
KangOxfordKangOxford

lustre-storage-audit

Audit Lustre storage usage by reconciling du results with lfs quota.

Community
Advanced
shakedfbshakedfb

seed

Seed source files into a processing queue with duplicate checks and archive management.

Community
Advanced
kim-emkim-em
112

lean-zstd-spec-pattern

Create formal Zstandard specifications in Lean with structured patterns.

Community
Advanced
serejarisserejaris
215

macos-fixer

Diagnose macOS memory pressure and output prioritized fixes.

Community
Basic