globus-transfer

Transfer directories between HPC clusters using globus-cli and Globus collections.

1|Updated Oct 25, 2020
One-click install
npx skills add https://github.com/bfairkun/dotfiles --skill globus-transfer-bfairkun
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: globus-transfer
Source: https://github.com/bfairkun/dotfiles/tree/main/agents/.agents/skills/globus-transfer
Command: npx skills add https://github.com/bfairkun/dotfiles --skill globus-transfer-bfairkun

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires globus-cli.

What problem does it solve? Moving large research data trees between HPC clusters (UChicago Midway, UMich Great Lakes) fails in non-obvious ways: missing consents, unlinked identities, hidden dotfiles, and transfers that silently retry until they expire. This Skill encodes the exact globus-cli setup, collection IDs, consent commands, and failure-mode diagnostics needed to complete cross-site transfers reliably. ## Core Features & Use Cases - Environment and account setup: Install globus-cli via mamba, link institutional identities, and grant data-access consents for every collection in one browser round-trip. - Batch-based transfers: Exclude large subdirectories precisely with batch files instead of error-prone --exclude filter rules, and submit resumable transfers with --sync-level mtime and --preserve-mtime. - Failure diagnosis: Interpret EXPIRED deadlines, LOGIN_DENIED, EndpointPermissionDenied, and zero-byte transfers using globus task event-list. - Use Case: You need to mirror a Midway3 project directory (including .git and .snakemake) to a UMich Turbo volume while skipping a huge Alignments folder; this Skill gives you the batch file, validation loop, and submission command. ## Quick Start Ask the AI to transfer a project directory from Midway3 to Great Lakes using globus-cli, excluding a specified large subdirectory.

Frequently Asked Questions about globus-transfer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I transfer files between HPC clusters with globus-cli?▼

Use globus transfer with source and destination collection IDs and paths, plus --batch for selective file lists and --sync-level mtime for resumable reruns. Both institutional identities must be linked in one Globus account first, and each collection needs a one-time data-access consent.

How to exclude subdirectories in a Globus transfer?▼

Use a batch file listing exactly what to move rather than --exclude filter rules, which match names at every depth. Each batch line is [--recursive] SOURCE DEST with paths relative to the endpoint paths, and you can validate paths locally before submitting.

Why does my Globus transfer expire with zero bytes transferred?▼

An EXPIRED deadline with 0 bytes is a symptom: Globus retried a failing subtask every ~5 minutes until the deadline. Run globus task event-list on the task ID to find the underlying AUTH or PERMISSION_DENIED events causing the retries.

Why does globus ls not show dotfiles like .git?▼

globus ls hides dotfiles by default; use globus ls --all to see .git, .gitignore, and .snakemake. Without --all, a correctly transferred tree can appear to be missing files.

What causes LOGIN_DENIED errors with the UMich Turbo collection?▼

The standalone UMich Turbo collection has been observed to reject logins outright with 530 Login incorrect errors. Switch to the umich#greatlakes collection, which mounts Turbo at /nfs/turbo/<volume>/, and prefix paths accordingly.

Does globus-cli require a browser for setup?▼

Yes, two one-time steps need a browser and cannot be automated from a login node: linking institutional identities at app.globus.org/settings/identities, and running globus login --no-local-server. Data-access consents also require one browser round-trip per session.