production-debugging

Diagnose production incidents by correlating logs, metrics, and distributed traces.

2|Updated Mar 15, 2025

Search Tags:#debugging #troubleshooting #production #logs #metrics #observability #incident response

Ready to run

npx skills add https://github.com/dandudzi/dotfiles --skill production-debugging-dandudzi

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: production-debugging
Source: https://github.com/dandudzi/dotfiles/tree/main/dot_claude/skills/production-debugging
Command: npx skills add https://github.com/dandudzi/dotfiles --skill production-debugging-dandudzi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a systematic approach to diagnosing and resolving complex issues that arise in live production environments, minimizing downtime and impact.

Core Features & Use Cases

Hypothesis-Driven Debugging: Guides users through a structured loop of forming, gathering evidence for, and validating hypotheses to pinpoint root causes.
Log, Metric, and Trace Analysis: Offers methods and examples for querying structured logs (LogQL, Elasticsearch DSL, CloudWatch Insights), correlating metrics, and analyzing distributed traces to understand system behavior.
Live Debugging & Profiling: Provides practical commands for live debugging in Kubernetes and Linux environments, along with tools for profiling CPU, memory, and goroutine leaks in various runtimes (Python, JVM, Go, Node.js).
Database & Rollback Guidance: Includes strategies for debugging database performance and lock contention, and a decision matrix for when to rollback a deployment.
Use Case: When a critical service experiences a sudden spike in error rates, this Skill can be activated to systematically analyze logs, metrics, and traces around the time of the incident, correlating the spike with recent deployments or configuration changes to identify the faulty component.

Quick Start

Activate the production-debugging skill to analyze recent errors by correlating logs and metrics from the last hour.

Frequently Asked Questions about production-debugging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I debug production incidents by correlating logs, metrics, and traces?▼

Debug production incidents by applying a hypothesis-driven debugging loop to correlate logs, metrics, and distributed traces. This methodology pinpoints root causes by validating system behavior evidence around the incident timeframe.

What is the best way to analyze live system performance in Kubernetes environments?▼

Analyze live system performance in Kubernetes environments using provided practical profiling commands. These tools help diagnose CPU, memory, and goroutine leaks across Python, JVM, Go, and Node.js runtimes directly in the cluster.

How do I query structured logs during incident response using LogQL or Elasticsearch DSL?▼

Query structured logs during incident response using provided methods and examples for LogQL, Elasticsearch DSL, and CloudWatch Insights. These querying languages help extract relevant error data to validate debugging hypotheses.

When should I rollback a deployment during a production incident?▼

Rollback a deployment during a production incident by consulting the included decision matrix. This framework evaluates database lock contention, performance degradation, and system impact to determine the exact rollback threshold.

Can I use this methodology to investigate database performance issues and lock contention?▼

Yes, you can investigate database performance issues and lock contention using the included strategies. These methods guide live system analysis to identify faulty components and resolve complex query bottlenecks in production.