wandb-experiment-tracking

What problem does it solve?

Ensures experiment logs, resolved configs, and system telemetry are consistently captured in Weights & Biases so runs are reproducible, debuggable from the dashboard, and comparable without SSH access to compute nodes.

Core Features & Use Cases

• Config and metadata capture: record the fully resolved experiment config at initialization and log derived values such as parameter counts and GPU model.
• Two-tier metric logging: separate batch-level metrics (loss, lr, throughput) from epoch-level summaries and always include explicit step and total iteration metadata.
• HPC and offline support: operate in online or WANDB_MODE=offline modes with clear offline sync workflows and Slurm-aware run naming for easy cross-referencing.
• System telemetry and console parity: log GPU memory_reserved to match nvidia-smi, ensure terminal logs appear in W&B Logs, and centralize wandb.log calls to avoid fragmentation.
• Ablation and comparison workflows: group related runs, prefix run names with versioning, and rely on the W&B Python API to programmatically compare runs and access git metadata.

Quick Start

Initialize a wandb run at program start, update wandb.config with the resolved experiment config, and log batch and epoch metrics with explicit step numbers so dashboards show complete, comparable runs.