system-profile

System profiling diagnoses GPU bottlenecks and memory redundancy by dynamically selecting external profiling tools and inserting code instrumentation to measure compute behavior, memory usage, and wall time. It generates a structured report with actionable recommendations saved to the profile_output directory.

Profiling a running process for CPU overhead and interconnect communication requires targeting the specific PID and applying minimal-overhead instrumentation to capture transfer latency, bandwidth, and collective communication characteristics. This produces a structured performance summary identifying hotspots and inefficiencies.

Code instrumentation measures bandwidth, transfers, and collective communication characteristics when external profiling tools are insufficient. All modified or created files are tracked in a mandatory instrumentation changelog to ensure transparent profiling artifact capture.

Profiling end-to-end framework execution identifies bottlenecks spanning multiple components by dynamically selecting appropriate external profiling tools and optional code instrumentation. It evaluates CPU overhead, memory usage, GPU compute behavior, and interconnect communication to produce a comprehensive performance report.

System profiling works with standalone scripts, GPU-focused workloads, running processes, and end-to-end framework execution. You initiate profiling by providing a clear target such as a script name, a PID, or a service descriptor like vllm serving.