debug-with-grafana

Follow the 7-step workflow: list datasource UIDs with gcx datasources list, confirm the service is scraped via the up metric, query the 5xx error rate over the incident window, check P95 latency, then pull matching error logs from Loki and review firing alert rules.

First identify the incident start time from the error-rate range query in Prometheus, then query Loki with a stream selector like {job="service"} |= "error" over the same window. Matching log timestamps to the metric spike reveals the failing code path or dependency.

Yes, gcx traces query accepts TraceQL expressions against a Tempo datasource UID. Attributes must be scoped, such as resource.service.name or span.http.status_code, and gcx traces get <trace-id> --llm retrieves a single trace in a token-efficient format.

Empty results usually mean the label selector matches no streams, the time range falls outside retention, or filters are too restrictive. Verify stream labels with gcx logs labels, broaden the time range, and simplify the selector to confirm data exists.

A 401 or 403 indicates an expired, missing, or under-privileged API token. Run gcx config view to inspect the active context, switch contexts with gcx config use-context if needed, and update the token via gcx config set before retrying.

Tempo requires scoped attribute names, so unscoped dotted names like service.name are rejected. Use resource.service.name for resource attributes and span.http.status_code for span attributes, or trace:rootService for root-span filtering.