observability-testing-patterns

Validate dashboards, alerting rules, logs, APM traces, and SLO compliance.

1|Updated Dec 29, 2025
One-click install
npx skills add https://github.com/aquariuscook/Agent_Modus_Map --skill observability-testing-patterns-aquariuscook
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: observability-testing-patterns
Source: https://github.com/aquariuscook/Agent_Modus_Map/tree/main/.claude/skills/observability-testing-patterns
Command: npx skills add https://github.com/aquariuscook/Agent_Modus_Map --skill observability-testing-patterns-aquariuscook

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the critical need to validate the accuracy, reliability, and performance of observability and monitoring systems, ensuring that dashboards, alerts, and data pipelines provide trustworthy insights.

Core Features & Use Cases

  • Dashboard Validation: Verifies that data displayed on dashboards (e.g., Kibana, Grafana) accurately reflects the source data.
  • Alerting Rule Testing: Ensures that alert rules trigger correctly at defined thresholds and recover as expected.
  • Log Aggregation Completeness: Confirms that logs from all services are captured and searchable in a centralized system.
  • APM Trace Validation: Checks that distributed traces capture the full request path and error information.
  • SLA/SLO Verification: Validates that key performance indicators meet defined Service Level Objectives.
  • Use Case: When a new dashboard is deployed, use this Skill to automatically verify that its key metrics match direct database queries and that its alert rules fire appropriately under simulated failure conditions.

Quick Start

Use the observability-testing-patterns skill to validate the accuracy of the 'operations-overview' dashboard and test its associated alert rules.

Frequently Asked Questions about observability-testing-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate that my Kibana dashboards accurately reflect source data?

Dashboard validation verifies that metrics displayed on Kibana or Grafana dashboards match direct database queries. This ensures your monitoring visualizations accurately represent the underlying source data without discrepancies.

How do I test alerting rules in Prometheus before they go live?

Alerting rule testing ensures your Prometheus rules trigger correctly at defined thresholds and recover as expected. You can validate alert firing and recovery behavior under simulated failure conditions before deployment.

What is the best way to verify distributed traces in Jaeger capture full request paths?

APM trace validation checks that distributed traces in Jaeger capture the complete request path and error information. This ensures your tracing infrastructure records accurate span data across all service boundaries.

Can I verify SLO compliance across multiple observability tools like Prometheus and PagerDuty?

SLA and SLO verification validates that key performance indicators meet defined Service Level Objectives. This works across various tools including Prometheus, Jaeger, and PagerDuty to ensure compliance monitoring is accurate.

Why does my alert rule fire incorrectly even when metrics appear normal on dashboards?

Incorrect alert firing often indicates a mismatch between dashboard data representation and alert rule thresholds. Testing alert rules under simulated failure conditions validates that thresholds align with actual metric behavior.