observability-k8s-investigation

Diagnose Kubernetes workload, node, and control-plane failures using OTel telemetry in Elasticsearch.

568|48|Updated Mar 6, 2026
One-click install
npx skills add https://github.com/elastic/agent-skills --skill observability-k8s-investigation
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: observability-k8s-investigation
Source: https://github.com/elastic/agent-skills/tree/main/plugins/observability/skills/k8s-investigation
Command: npx skills add https://github.com/elastic/agent-skills --skill observability-k8s-investigation

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Kubernetes incidents like CrashLoopBackOff, OOMKilled pods, stuck rollouts, and node pressure are hard to root-cause because signals are scattered across metrics, events, logs, and APM. This Skill gives an AI agent a structured investigation methodology to correlate OTel telemetry collected via EDOT and pinpoint the failure mode with calibrated confidence.

Core Features & Use Cases

  • Failure-mode taxonomy: Classifies incidents across workload, node, control-plane, autoscaling, and networking layers using pivotal signals like k8s.container.status.last_terminated_reason and k8s.pod.cpu_limit_utilization.
  • Guided investigation flow: Walks through orient, characterize, classify, corroborate, and synthesize phases with ready-to-use ES|QL query recipes against OTel receiver indices.
  • Confidence-calibrated synthesis: Produces structured incident reports with hypothesis, evidence, confidence level, and recommended next steps, explicitly avoiding confabulation from empty results.
  • Use Case: When a pod alert fires for CrashLoopBackOff, the agent checks restart counts and termination reasons, distinguishes OOMKilled from CPU throttling from probe misconfiguration, corroborates with K8s events and baselines, and delivers a root-cause hypothesis.

Quick Start

Ask the agent to investigate why a specific pod in a given namespace is CrashLoopBackOff-ing using the Kubernetes investigation skill.

Frequently Asked Questions about observability-k8s-investigation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I investigate a CrashLoopBackOff pod in Kubernetes?

Start by checking k8s.container.restarts and k8s.container.status.last_terminated_reason in the k8sclusterreceiver index. OOMKilled points to the memory path, Error with high CPU utilization points to throttling, and ContainerCannotRun points to image or exec issues.

How to detect CPU throttling in Kubernetes pods with OpenTelemetry?

Query k8s.pod.cpu_limit_utilization in metrics-kubeletstatsreceiver.otel-* and check max and p95 values, not averages. A sustained ratio above 1.0 indicates CFS throttling, which commonly causes liveness probe timeouts and Error-reason terminations.

Does this skill work with the legacy Elastic Kubernetes integration?

No, it only supports the OTel ingest path via EDOT and kube-stack collector indices. The legacy metrics-kubernetes.* and logs-kubernetes.* data streams are deprecated and explicitly out of scope.

Why does my Kubernetes investigation show no logs for a failing pod?

Empty log results usually mean logs are not collected or the pod has no recent lines, not a specific failure mode. The skill reports no_logs_available and weights remaining signals like events and metrics instead of inventing a cause.

What ES|QL gotchas affect Kubernetes observability queries?

VALUES() returns a scalar for single values, PERCENTILE does not work on OTel histogram types, and COUNT on aggregate_metric_double returns value_count rather than document count. K8s metrics require flat field paths like k8s.pod.name in ES|QL.

When should I not trust the k8s.container.restarts counter?

The restart count is pulled from the K8s API and may be pruned by the kubelet, so treat it as boolean: zero versus greater than zero. Confirm actual restart patterns through Killing and BackOff events in the k8seventsreceiver index.