setup-sunk-gpu-monitoring

To set up GPU and Slurm metrics collection on a SUNK cluster, use this Skill to deploy DCGM Exporter and SUNK Syncer for Prometheus scraping. It configures data gathering for GPU utilization and Slurm scheduler health.

Deploying GPU monitoring requires an existing SUNK cluster with active GPU compute nodes and a dedicated monitoring namespace. These prerequisites ensure the DCGM Exporter and SUNK Syncer components can successfully scrape and aggregate metrics.

Yes, SUNK GPU monitoring is provider-agnostic and works across different cloud providers, including GKE, EKS, and bare-metal deployments. It consistently gathers GPU and Slurm metrics regardless of the underlying infrastructure.

Yes, you can visualize GPU utilization and Slurm metrics using optional Grafana dashboards. Once metrics are scraped by Prometheus, Grafana integrates to display temperature, power, and scheduler performance data.

DCGM Exporter integrates with Prometheus by exposing detailed GPU metrics like temperature and power for scraping. This Skill configures the SUNK Syncer to bridge these metrics into your Prometheus monitoring namespace.

The best way to monitor Slurm node and scheduler metrics on Kubernetes is using a SUNK cluster setup with SUNK Syncer. It captures Slurm job data and feeds it into Prometheus for comprehensive cluster health observability.