perf-workload-profiling

To measure PyTorch training loop latency and throughput accurately, inject manual timing to capture per-iteration latency, compute throughput in samples/sec, and record dataloader time using CUDA synchronization. Proper instrumentation requires warmup exclusion and pre-allocation to avoid perturbing results.

The best way to benchmark single CUDA operations is using CUDA events with warmup iterations and per-iteration statistics like mean, median, std, min, and max. This approach avoids measurement-loop anti-patterns and synchronization issues that can skew kernel timing results.

NVTX annotations improve GPU profiling timeline readability by adding human-readable range labels to the profiler output. You apply these annotations via decorators or context managers to clearly mark execution phases, though NVTX should not be used for actual performance measurement.

PyTorch GPU timing measurements are inconsistent when warmup iterations are not excluded, memory is not pre-allocated, or when synchronization anti-patterns occur. Accurate GPU profiling requires disciplined warmup exclusion, pre-allocation, and proper CUDA event usage to avoid perturbing results.

You can use synchronized host timing instead of CUDA events for GPU profiling in PyTorch, provided you apply strict synchronization discipline. Both methods achieve GPU-accurate timing when paired with warmup exclusion and pre-allocation to prevent measurement perturbation.

When profiling a PyTorch training loop, track per-iteration latency, throughput in samples/sec, and dataloader or data-load time. These metrics, captured with warmup exclusion and GPU-accurate timing, reveal workload bottlenecks and true training performance.