graphsignal-profiler

You can profile LLM inference latency without modifying source code by running the graphsignal-run command followed by your inference server launch command to automatically instrument your workload through a sidecar process.

GPU monitoring supports major inference frameworks including vLLM, SGLang, and PyTorch across various hardware accelerators to capture hardware-level metrics and identify underutilized resources.

OpenTelemetry tracing for AI inference records high-resolution timelines for operation durations and LLM generation steps, enabling automated tracing of inference operations to pinpoint latency bottlenecks.

You can identify underutilized GPU resources during model deployment by capturing system-level telemetry and hardware-level metrics for CPUs and GPUs to monitor device utilization and failures.

The best way to monitor production inference workloads for bottlenecks is through a sidecar process that observes operations, providing performance profiling and resource utilization metrics without code changes.

Inference tracing with OpenTelemetry requires specific dependencies including opentelemetry-proto, grpcio, nvidia-ml-py, and prometheus-client to capture system-level telemetry and hardware metrics.