observability-and-instrumentation

Instruments production code with structured logs, metrics, traces, and symptom-based alerts.

1|Updated Mar 2, 2025
One-click install
npx skills add https://github.com/marjorg/setup --skill observability-and-instrumentation-marjorg
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: observability-and-instrumentation
Source: https://github.com/marjorg/setup/tree/main/home/.agents/skills/observability-and-instrumentation
Command: npx skills add https://github.com/marjorg/setup --skill observability-and-instrumentation-marjorg

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Production features often ship without telemetry, so when something breaks, engineers cannot tell what happened from the available data. This Skill guides you to instrument code alongside the feature so production behavior is visible, queryable, and diagnosable from day one. ## Core Features & Use Cases - Structured Logging: Enforces JSON log events with stable event names, correlation IDs, entry-point attribution, and consistent log levels, while blocking secrets and PII from log output. - Metrics and Tracing: Applies RED/USE metric patterns with bounded label cardinality, histogram-based latency percentiles, and OpenTelemetry distributed tracing with context propagation. - Alerting and Runbooks: Designs symptom-based alerts with two severity tiers, actionable thresholds, and linked runbooks, plus a verification pass that test-fires alerts and validates telemetry end-to-end. - Use Case: When adding a payment retry flow with a queue and an external provider, use this Skill to define the on-call questions, add structured payment_failed events, RED metrics on the provider calls, a trace across services, and a symptom alert with a runbook before shipping. ## Quick Start Ask the AI to instrument your new production endpoint or background job with structured logging, RED metrics, tracing, and an alert with a runbook using this observability skill.

Frequently Asked Questions about observability-and-instrumentation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add structured logging to a production service?

Structured logging means emitting JSON log events with stable event names and machine-readable fields instead of interpolated strings. Attach a correlation ID at the system boundary, propagate it to every log line and outbound call, and never log secrets or full PII.

What metrics should I add to a new API endpoint?

Use the RED pattern: rate, errors, and duration as a latency histogram with p95/p99 percentiles. Keep label cardinality bounded to small fixed sets like route template and status class, and never use user IDs or raw URLs as labels.

Should I use OpenTelemetry or Prometheus for instrumentation?

OpenTelemetry is the vendor-neutral standard for tracing and metrics with auto-instrumentation for HTTP, gRPC, and common database clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Why do high-cardinality metric labels cause problems?

Every unique label combination creates a separate time series, so unbounded values like user IDs, request IDs, or error messages explode storage and query costs in the metrics backend. High-cardinality lookups belong in logs and traces instead.

When should an alert page someone versus create a ticket?

Page only for user-facing symptoms like elevated error rate or latency that require immediate action, and use tickets for degradations that can wait. Alerts on causes like CPU or disk belong on dashboards, not on the pager.

When is this observability approach not the right tool?

It is not for diagnosing an active failure, which belongs to debugging workflows, nor for profiling measured slowness, which belongs to performance optimization. It covers the instrumentation written alongside features that makes those later investigations fast.