gpu-train-monitor

Monitors GPU health and summarizes training progress from live system data.

2|Updated Mar 20, 2026

Search Tags:#gpu #throughput #log-analysis #wandb #nvidia-smi #training-monitoring #multigpu

Ready to run

npx skills add https://github.com/Gonglitian/agent-skills --skill gpu-train-monitor

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: gpu-train-monitor
Source: https://github.com/Gonglitian/agent-skills/tree/main/skills/gpu-train-monitor
Command: npx skills add https://github.com/Gonglitian/agent-skills --skill gpu-train-monitor

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps ML engineers and AI researchers quickly assess and diagnose multi-GPU training jobs by aggregating GPU status, training progress, and log data into actionable insights.

Core Features & Use Cases

GPU status & utilization: snapshot GPUs' usage, memory, temperature to prevent bottlenecks.
Training progress & logs: parse active runs to extract epoch/step, loss trends, learning rate, and throughput.
Integrated monitoring: support WandB integration detection and produce concise status reports for team standups.
Use Case: A data scientist monitors a multi-node training job and receives a summary when the job stalls or high memory usage is detected.

Quick Start

Ask the skill to generate a real-time status report for your current multi-GPU training job.

Frequently Asked Questions about gpu-train-monitor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor multi-GPU training status and utilization in real-time?▼

To monitor multi-GPU training, you need a tool that snapshots GPU utilization, memory, and temperature for all active GPUs. It should parse active training processes, map them to scripts and config files, and summarize training progress, loss trends, and throughput into a concise status report.

Can I integrate WandB logs when monitoring my active training processes?▼

Yes, training monitoring can optionally integrate WandB when it is configured. The monitoring process parses active runs to extract epoch/step data, loss trends, learning rate, and throughput, combining WandB metrics with local log data for comprehensive reporting.

What is the best way to diagnose a stalled multi-GPU training job with high memory usage?▼

Diagnosing a stalled multi-GPU training job requires aggregating GPU status, training progress, and log data. By snapshotting usage, memory, and temperature, then extracting loss trends and throughput, you receive actionable insights and guardrails to improve performance and stability.

How does GPU training monitoring extract learning rate and throughput from active runs?▼

GPU training monitoring extracts learning rate and throughput by parsing active training processes and locating their corresponding log files. It maps these processes to their original scripts and config files to summarize the exact training progress and trends.

Does multi-GPU monitoring work without installing additional dependencies or components?▼

Yes, multi-GPU monitoring operates with zero dependencies or components. It directly identifies and reports the current GPU status, including utilization, memory usage, and temperature, by parsing active runs and their native logs on the system.