using-the-filesystem

Name: using-the-filesystem
Availability: InStock
Author: yale-som-hpc

Guide data placement and write patterns on GPFS for HPC environments.

5|1|Updated Apr 29, 2026

Search Tags:#filesystem #hpc #data-management #scratch #atomic-writes #gpfs

Ready to run

npx skills add https://github.com/yale-som-hpc/claude-code-marketplace --skill using-the-filesystem

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: using-the-filesystem
Source: https://github.com/yale-som-hpc/claude-code-marketplace/tree/main/plugins/hpc/skills/using-the-filesystem
Command: npx skills add https://github.com/yale-som-hpc/claude-code-marketplace --skill using-the-filesystem

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

GPFS metadata is shared and can become a bottleneck when many small files are created or when writes are not carefully staged. This Skill guides you to place data in the right GPFS locations and to use safe write patterns to minimize metadata storms and contention.

Core Features & Use Cases

Guidance on where to put data: personal home, shared project space, and scratch.
Best practices for atomic writes and avoiding huge numbers of small files.
Practical workflows for organizing project data, scratch, and temporary work during HPC jobs.

Quick Start

Organize your workflow by placing shared data under /gpfs/project, use /tmp for temporary compute tasks, and adopt atomic writes before finalizing outputs.

Frequently Asked Questions about using-the-filesystem

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is the best way to organize GPFS storage across home, project, and scratch spaces on an HPC cluster?▼

The best way to organize GPFS storage is to place shared project data under /gpfs/project, keep personal files in /gpfs/home, and use scratch space for temporary work during jobs to minimize metadata contention.

How do I avoid GPFS metadata bottlenecks when writing large numbers of small files?▼

You can avoid GPFS metadata bottlenecks by adopting atomic write patterns, staging temporary outputs in /tmp, and finalizing results to shared locations only when ready, which reduces metadata storms from many small file operations.

Why does GPFS performance drop during multi-user HPC jobs with heavy file operations?▼

GPFS performance drops because metadata is shared across nodes and becomes a bottleneck when many users create small files or perform uncoordinated writes, causing contention that slows down overall filesystem throughput.

Can I use /tmp for temporary compute tasks instead of GPFS scratch space?▼

Yes, you can use /tmp for temporary compute tasks during active jobs. It handles node-local temporary work efficiently, while GPFS scratch space is better suited for larger intermediate data that needs to persist across submission nodes.

When should I use atomic writes for HPC data management on a GPFS filesystem?▼

You should use atomic writes whenever finalizing output files on GPFS, especially in multi-user environments. This write pattern prevents partial writes and minimizes metadata overhead by staging data before committing it.

Does atomic write guidance apply to both personal home and shared project data on GPFS?▼

Yes, atomic write guidance applies to both home and project spaces. Using safe write patterns across all GPFS locations ensures data integrity and prevents metadata contention regardless of whether the target is personal or shared.