observability-and-instrumentation

Instrument production code with structured logging, RED metrics, and OpenTelemetry tracing.

Updated Nov 24, 2025
One-click install
npx skills add https://github.com/ByronWilliamsCPA/.claude --skill observability-and-instrumentation-byronwilliamscpa
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: observability-and-instrumentation
Source: https://github.com/ByronWilliamsCPA/.claude/tree/main/.claude/skills/observability-and-instrumentation
Command: npx skills add https://github.com/ByronWilliamsCPA/.claude --skill observability-and-instrumentation-byronwilliamscpa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Unstructured, missing, or irrelevant telemetry turns production incidents into time-consuming archaeology, leaving teams unable to quickly diagnose failures or verify that new features work as expected in production.

Core Features & Use Cases

  • Structured Logging & Correlation IDs: Emit queryable JSON log events with mandatory correlation IDs to reconstruct full request paths across distributed services, with strict rules to avoid logging secrets or PII.
  • RED/USE Metrics: Instrument endpoints and dependencies with rate, error, and duration (or utilization, saturation, error) metrics using bounded label sets to avoid cardinality issues, with percentile-based latency tracking instead of averages.
  • Distributed Tracing & Alerting: Implement cross-service tracing with OpenTelemetry and set up symptom-based alerts that fire only when users are impacted, linked to runbooks for fast on-call response.
  • Use Case: Use this skill when shipping a new payment processing feature to add the right telemetry so on-call engineers can diagnose payment failures in minutes instead of hours.

Quick Start

Use the observability-and-instrumentation skill to add structured logging, RED metrics, and symptom-based alerting to the new user authentication endpoint before it is deployed to production.

Frequently Asked Questions about observability-and-instrumentation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add structured logging and correlation IDs to distributed services?

Structured logging with correlation IDs involves emitting queryable JSON log events to reconstruct request paths across services, ensuring no secrets or PII are logged. This enables full visibility into system behavior and fast incident diagnosis.

What's the best way to instrument RED metrics without causing cardinality issues?

To prevent metric cardinality issues, instrument RED (rate, error, duration) metrics using bounded label sets and track percentile-based latency instead of averages. This keeps telemetry data actionable while monitoring endpoint performance.

How does symptom-based alerting work for production incidents?

Symptom-based alerting fires only when users are impacted, reducing alert fatigue. Alerts are linked to runbooks for fast on-call response, ensuring observability data drives immediate and actionable incident diagnosis rather than noise.

When should I set up distributed tracing with OpenTelemetry?

Set up distributed tracing with OpenTelemetry when shipping production-facing features, services, or external integrations. Cross-service tracing enables full visibility into system behavior and allows on-call engineers to diagnose failures in minutes.

How do I validate telemetry data is secure and actionable before deployment?

Validate telemetry data by applying strict rules to avoid logging secrets or PII, ensuring structured logs and metrics are queryable. Telemetry validation ensures observability data remains actionable and secure for production incident diagnosis.

Do I need observability instrumentation for background jobs and endpoints?

Yes, observability instrumentation applies to all production-facing features, endpoints, background jobs, and external integrations. Emitting structured telemetry for these components ensures fast diagnosis of failures and verifies production behavior.