tikv-performance

Diagnose TiKV latency spikes, ServerIsBusy errors, and IOPS saturation.

Updated Jun 3, 2026
One-click install
npx skills add https://github.com/alex-quan-001/nutshell-skills --skill tikv-performance
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: tikv-performance
Source: https://github.com/alex-quan-001/nutshell-skills/tree/main/skills/diagnosis/tikv-performance
Command: npx skills add https://github.com/alex-quan-001/nutshell-skills --skill tikv-performance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates the guesswork in troubleshooting TiKV performance issues that lack obvious crash symptoms, such as unexplained latency spikes, ServerIsBusy errors, disk IOPS saturation, and flow-control throttling, by providing structured, battle-tested diagnostic guidance for on-call engineering teams.

Core Features & Use Cases

  • gRPC Latency Diagnosis: Identify and resolve TiDB-to-TiKV RPC long-tail latency caused by thread hotspots, compression bottlenecks, or connection imbalances.
  • Disk & IOPS Troubleshooting: Address TiKV disk IOPS exhaustion triggered by internal SQL (e.g., ANALYZE), large cold-data scans, or disk space exhaustion.
  • Flow Control & Throttling Resolution: Diagnose write-surge-induced throttling, level-0 SST file accumulation, and scheduler-reported ServerIsBusy errors.
  • Large Region & Hotspot Mitigation: Fix oversized regions from hot-row updates, load-based split failures, and scheduling-induced performance degradation.
  • Use Case: For example, if your TiDB cluster is experiencing unexplained p99 latency spikes with ServerIsBusy errors during normal traffic, use this Skill to quickly rule out gRPC bottlenecks, check for internal SQL-induced IOPS saturation, and identify if flow control or large regions are the root cause.

Quick Start

Use the tikv-performance skill to diagnose the root cause of unexplained TiKV latency spikes and ServerIsBusy errors in your current cluster incident.

Frequently Asked Questions about tikv-performance

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot unexplained TiKV latency spikes and ServerIsBusy errors?

To troubleshoot TiKV latency spikes and ServerIsBusy errors, check for gRPC thread hotspots, disk IOPS exhaustion from internal SQL, or flow-control throttling. Apply metric checklists to quickly isolate whether the bottleneck is connection imbalance or scheduler saturation.

What causes TiKV flow control throttling and how can I resolve it?

TiKV flow control throttling is typically caused by write surges and level-0 SST file accumulation. You can resolve this performance issue by diagnosing the write load, checking scheduler backlogs, and applying workarounds to alleviate the immediate disk pressure and stabilize cluster throughput.

How do I fix TiKV disk IOPS exhaustion during cold-data scans or internal SQL operations?

Fix TiKV disk IOPS exhaustion by identifying internal SQL commands like ANALYZE or large cold-data scans consuming disk bandwidth. Apply diagnostic guidance to throttle or reschedule these operations, preventing resource saturation and restoring normal IOPS availability for TiKV.

Why does a large region cause performance degradation in TiKV and how do I mitigate it?

A large region causes TiKV performance degradation through hot-row updates and load-based split failures. Mitigate this scheduling side effect by identifying oversized regions and applying targeted split strategies to balance the load and eliminate the resulting RPC long-tail latency.

Does this diagnostic guidance apply to on-call scenarios involving TiDB-to-TiKV RPC long-tail latency?

Yes, this guidance applies to on-call engineering scenarios involving TiDB-to-TiKV RPC long-tail latency. It provides battle-tested workarounds and version-specific known issue references to accelerate root cause identification for unexplained performance bottlenecks without obvious crash symptoms.

What are the limitations when diagnosing TiKV performance bottlenecks without obvious crash symptoms?

Diagnosing TiKV performance bottlenecks without crash symptoms is limited by the need to correlate multiple metrics like gRPC latency, disk IOPS, and flow control. Complex interactions like scheduling-induced side effects require structured checklists to avoid missing subtle root causes.