troubleshoot-zymtrace-backend

To troubleshoot missing zymtrace profiling data on Kubernetes, diagnose the end-to-end ingest pipeline by checking profiler-agent connectivity, ingest pod health, and ClickHouse PVC storage capacity to isolate the broken link.

The zymtrace ingest pod may crash loop due to ClickHouse connectivity issues, license/auth errors, or storage full-disk pressure. Gather helm/kubectl state and logs to diagnose backend ingest processing failures and resolve dependencies.

Fix ClickHouse connectivity issues by checking PVC health, storage capacity, and full-disk pressure that prevent zymtrace profile ingestion from persisting. Diagnose ClickHouse state alongside helm/kubectl logs to identify and resolve storage exhaustion.

Yes, the zymtrace troubleshooting process supports GPU CUDA injection diagnostics by gathering and interpreting profiler-agent logs for CUDA injection signals in Kubernetes Helm installations. This helps validate profiler reporting for GPU workloads.

If zymtrace shows no data after a Helm upgrade, verify profiler-agent reporting, check ingest pod health for CrashLoopBackOff, and validate ClickHouse storage. Apply proper reset semantics during Helm upgrades and rerun verification steps.

The safety limitation when diagnosing zymtrace backend failures is avoiding destructive actions without explicit confirmation. The troubleshooting process enforces guardrail constraints, requiring verification-first fixes and explicit approval before capacity expansion or retention adjustments.