Diagnose training incidents by querying per-job Prometheus TSDB metrics.
npx skills add https://github.com/AMD-AGI/maxtext-slurm --skill tsdb-diagnosis
Please help me install this Agent Skill. Skill: tsdb-diagnosis Source: https://github.com/AMD-AGI/maxtext-slurm/tree/main/skills/tsdb-diagnosis Command: npx skills add https://github.com/AMD-AGI/maxtext-slurm --skill tsdb-diagnosis
Diagnose training incidents and assess cluster health by querying per-job Prometheus TSDB to identify root causes and validate monitoring data.
Run tsdb-diagnosis on a live or completed RAY=1 job to begin per-job TSDB analysis and triage.
High-intent search queries and answers about installing and using this skill.
Diagnose training incidents by querying per-job Prometheus TSDB to identify root causes and health issues. The skill correlates GPU, network, and system metrics with training events to validate monitoring data for live or completed RAY=1 jobs.
Yes, tsdb-diagnosis supports post-mortem analysis for completed RAY=1 jobs. You can query the per-job TSDB directory to review historical Prometheus metrics, parse TensorBoard metrics, and align them with training steps to identify past root causes.
Troubleshooting Ray training jobs involves correlating GPU, network, and system metrics with training events. The skill aligns these Prometheus TSDB metrics with training steps via tb_step to provide triaged, step-aligned verification for accurate triage decisions.
Yes, you need access to the per-job TSDB directory to run tsdb-diagnosis. Requirements include the ability to read Prometheus metrics, parse TensorBoard metrics, and align metrics with training steps via tb_step to accurately diagnose training incidents.
Training job health assessment requires step-aligned metrics to accurately map TSDB data to training steps via tb_step. This step-aligned verification validates monitoring data and ensures triage decisions are based on precise correlations between metrics and training events.
The best way to triage GPU and network issues during live Ray training is to query per-job Prometheus TSDB data. This skill applies real-time troubleshooting to live RAY=1 jobs, using TSDB as the primary evidence source to identify root causes and health issues.
n4vyshMonitor cloud-native systems with Prometheus and Grafana Cloud Metrics.
multigresQuery the multigres observer API to diagnose cluster health and produce a structured bug report.
zhuangbiaoweiVisualize machine learning training metrics and model graphs with TensorBoard.
yaacovGuide Prometheus/Thanos cluster monitoring with kubectl-metrics queries and PromQL.
