host-health-diagnose

Diagnoses host unreachability by correlating heartbeat, metrics, and neighbor evidence.

13.3k|1.8k|Updated Mar 3, 2020
One-click install
npx skills add https://github.com/ccfos/nightingale --skill host-health-diagnose
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: host-health-diagnose
Source: https://github.com/ccfos/nightingale/tree/main/aiagent/skill/embedded/builtin/host-health-diagnose
Command: npx skills add https://github.com/ccfos/nightingale --skill host-health-diagnose

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a monitoring system marks a machine as unreachable, it is often unclear whether the host is truly down, the agent is hung, the network is flapping, or the machine is simply under maintenance. Concluding "down" from a single heartbeat signal is a common source of false-positive alerts and wasted on-call effort.

Core Features & Use Cases

  • Three-Layer Evidence Collection: Gathers real-time heartbeat status from Redis, a 10-minute metric window from the Prometheus datasource, and the health of neighbor hosts in the same business group.
  • Three-State Decision Table: Classifies the situation into truly down, agent hung, network flapping, under maintenance, cluster event, or insufficient data, each with a confidence level.
  • False-Positive Guardrails: Checks alert mutes, companion alerts, and clock skew before recommending any action, and always includes a self-verification step such as checking systemctl status categraf.
  • Use Case: A user asks "why is machine web-07 unreachable?" The skill pulls heartbeat lag, recent CPU/network metrics, and neighbor status, then concludes whether it is a single-host outage or a cluster-level event with concrete recommended actions.

Quick Start

Ask the assistant why a specific host shows as unreachable in Nightingale and whether the alert is a false positive.

Frequently Asked Questions about host-health-diagnose

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if a host-unreachable alert is a false positive?

Cross-validate the heartbeat status against the recent metric window and neighbor hosts. If metrics are still flowing while the heartbeat is stale, the agent is likely hung or the Redis heartbeat channel is lagging, not a real outage.

Why does Nightingale show a host as down when I can still ping it?

A pingable host with a stale heartbeat usually means the categraf agent is hung, cannot write heartbeats to Redis, or the network path to the server is partitioned. Check the metric stream's last timestamp to see if data is still flowing.

What does it mean when all hosts in a business group go stale at once?

Simultaneous staleness across a business group indicates a cluster-level event such as a switch failure, data center issue, or Redis write blocking under high server load. Investigate the shared infrastructure rather than individual machines.

Can a muted host still trigger unreachable alert investigations?

Yes, but the diagnosis checks alert mute rules first. If a mute matches, the host is under maintenance and the alert is not a false positive; the result includes when the mute window ends.

How do I verify whether the categraf agent is really dead?

SSH into the host and run systemctl status categraf, ps -ef | grep categraf, or journalctl -u categraf --since "5 min ago". Compare the result with the last reported cpu_util and heartbeat lag to confirm the diagnosis.