alert-rule-troubleshoot

Follow the data flow in order: verify the rule is enabled and correctly configured, confirm the datasource is bound to an alert engine cluster, run the rule's query manually, then read the engine evaluation logs with get_alert_eval_logs. The logs directly show whether data was found, conditions were met, or the event was muted.

Pull the event's processing logs with get_event_processing_logs using the event hash, then verify each bound notify rule with get_notify_rule_detail. Common causes are a disabled notify rule, severities not including the event level, time windows not covering the trigger moment, or rate limiting via notify_repeat_step and notify_max_number.

This workflow requires Nightingale Release 22 or above, because it depends on the R22+ engine evaluation logs and event processing logs exposed through the built-in MCP endpoint. Releases R21 and earlier are out of scope.

In ES query_string syntax, AND, OR, and NOT must be uppercase to act as boolean operators; lowercase and/or/not are treated as ordinary terms, changing the query semantics. Rewrite the operators in uppercase or use structured bool.must/bool.should clauses.

Multiple queries are grouped by fully identical label sets before cross-reference expressions are evaluated, so queries A and B must produce series with matching labels. Also, a ref used only in the recovery expression never fires alerts and can make recovery unsatisfiable; it must appear in the trigger expression.

Use alert rule troubleshooting when an expected alert did not fire or no notification arrived. Use ops-troubleshooting for the opposite direction: you already have an alert event and want to find the underlying root cause of the incident.