observability-and-instrumentation

Instruments production code with structured logs, metrics, traces, and symptom-based alerts.

1|Updated Sep 4, 2026
One-click install
npx skills add https://github.com/SanHsien/agent-skills --skill observability-and-instrumentation-sanhsien
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: observability-and-instrumentation
Source: https://github.com/SanHsien/agent-skills/tree/main/skills/observability-and-instrumentation
Command: npx skills add https://github.com/SanHsien/agent-skills --skill observability-and-instrumentation-sanhsien

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Production features often ship without telemetry, so when something breaks, engineers cannot tell what happened from the available data. This Skill guides you to instrument code alongside the feature itself, so every production behavior is visible, queryable, and diagnosable before the first incident. ## Core Features & Use Cases - Structured Logging: Enforces JSON log events with stable event names, mandatory correlation IDs, entry-point attribution, and strict rules against logging secrets or PII. - Metrics and Tracing: Applies RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) methods with bounded label cardinality, histogram-based latency percentiles, and OpenTelemetry distributed tracing. - Symptom-Based Alerting and Runbooks: Defines alerts on user-facing symptoms rather than causes, requires every alert to link a runbook, and includes a verification checklist to test-fire alerts and validate telemetry end-to-end. - Use Case: When adding a payment retry flow with an external provider, use this Skill to define the on-call questions first, then add structured payment_failed log events, a latency histogram for provider calls, a trace across services, and an alert on elevated error rate with a linked runbook. ## Quick Start Use the observability-and-instrumentation skill to add structured logging, RED metrics, tracing, and an alert with a runbook to my new checkout endpoint.

Frequently Asked Questions about observability-and-instrumentation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add structured logging to a production service?

Emit JSON log events with a stable event name and machine-readable fields instead of string interpolation. Attach a correlation ID generated at the system boundary to every log line, and never log secrets, tokens, or full request bodies.

What metrics should I track for an HTTP endpoint?

Use the RED method: request rate, error rate, and duration as a latency histogram with p50/p95/p99 percentiles. Keep label cardinality bounded to small fixed sets like route template and status class, never user IDs or raw URLs.

Should I use OpenTelemetry or Prometheus for instrumentation?

OpenTelemetry is the vendor-neutral standard for tracing and metrics with auto-instrumentation for HTTP, gRPC, and common DB clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Why are high-cardinality metric labels a problem?

Every unique label combination creates a separate time series, so labels like user IDs, raw URLs, or error messages make the metrics backend fall over. High-cardinality lookups belong in logs and traces, not metric labels.

When should an alert page someone versus create a ticket?

Page only for user-facing symptoms like error rate above 1% or p99 latency breaches that require immediate action. Use tickets for degradations that can wait a week, and alert on symptoms users feel rather than causes like CPU usage.

When should I not use this observability skill?

Do not use it to diagnose an active failure; use a debugging skill for that. It also does not cover performance profiling of measured slowness or launch-day monitoring checklists, which belong to separate skills.