skills.rest
mysticaltech
mysticaltechCOMMUNITY
Explore all skills by this author →
𝕏@mysticaltechGitHub

debug-node

Diagnose unreachable Hetzner Cloud nodes by mounting their filesystems in rescue mode.

3.9k|563|Updated Jul 30, 2021
Search Tags:#kubernetes#Troubleshooting#Hetzner#cloud-init#btrfs#rescue mode#ssh debugging
One-click install
npx skills add https://github.com/mysticaltech/terraform-hcloud-kube-hetzner --skill debug-node
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debug-node
Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/tree/main/.claude/skills/debug-node
Command: npx skills add https://github.com/mysticaltech/terraform-hcloud-kube-hetzner --skill debug-node

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a Hetzner Cloud server becomes unreachable—SSH hangs, provisioning stalls, or cloud-init fails—there is no way to inspect the node through normal access. This Skill boots the server into Hetzner rescue mode and mounts its disk so you can inspect logs, SSH keys, sshd config, and cloud-init state without needing working SSH on the node itself.

Core Features & Use Cases

  • Rescue-mode filesystem inspection: Enables rescue mode via the hcloud CLI, mounts the btrfs snapshot layout of openSUSE Leap Micro, and locates the active snapshot where /etc lives.
  • Structured diagnostic checklist: Walks through cloud-init results, authorized_keys, sshd config precedence, root account lock status, journal logs, NetworkManager profiles, k3s/RKE2 config, and SELinux denials.
  • Guided fixes and recovery: Applies targeted repairs such as unlocking the root account, restoring authorized_keys, and regenerating SSH host keys, then reboots back to normal mode.
  • Use Case: A Terraform provisioner hangs on "Still creating" because SSH never connects. Use this Skill to boot the node into rescue mode, discover that cloud-init never injected the SSH key, fix authorized_keys, and reboot into a working node.

Quick Start

Ask the assistant to debug an unreachable Hetzner server by providing its name or IP from hcloud server list and describing the symptom you are seeing.

Frequently Asked Questions about debug-node

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a Hetzner Cloud server when SSH is not working?▼

Enable rescue mode with hcloud server enable-rescue, reboot, then SSH into the rescue system and mount the server's disk. From there you can read cloud-init logs, authorized_keys, sshd config, and journal files without needing SSH on the node itself.

How do I mount a Leap Micro btrfs filesystem in rescue mode?▼

Mount with subvolid=5 to reach the btrfs top level, then navigate to @/.snapshots/ where the highest-numbered snapshot is the active one containing /etc. Logs and cloud-init state live separately under @/var, and /root is at @/root.

Why does SSH fail with "Too many authentication failures" on kube-hetzner nodes?▼

Kube-hetzner sets MaxAuthTries 2, so an SSH agent offering many keys gets disconnected before the right key is tried. Use ssh -o IdentitiesOnly=yes -i with the specific private key to avoid agent key spray.

Why is the root account locked on a fresh Hetzner node?▼

A locked root account shows as root:! or root:!* in /etc/shadow, and PAM rejects SSH even with a valid key when UsePAM is enabled. The fix is usermod -p '*' root inside the packer transactional-update snapshot, with a cloud-init bootcmd as a safety net.

Can I inspect journal logs from a server that will not boot?▼

Yes, use journalctl -D pointing at the mounted journal directory, for example journalctl -D /mnt/@/var/log/journal/ -u sshd. This reads logs offline without a running system, covering sshd, k3s, and rke2 services.

What are the limitations of fixing a node through rescue mode?▼

Rescue-mode edits are immediate manual fixes and do not persist into future node rebuilds. The durable fix belongs in the packer template or cloud-init configuration, and packer snapshots must be rebuilt so new nodes get the correction.

Related Skills

Explore More Software Engineering Skills→
evolv3aievolv3ai
3

admin-infra-hetzner

Automate Hetzner Cloud server provisioning with configurable types and locations.

Community
Advanced
drewid74drewid74
8

proxmox-k3s-infra

Provision and troubleshoot Proxmox homelab infrastructure for K3s Kubernetes clusters.

Community
Advanced
Sir-chawakornSir-chawakorn
9

k8s-debug-workload

Diagnose Kubernetes workload failures by analyzing pod status, events, logs, and node constraints.

Community
Advanced
ivegamsftivegamsft

docker-reset

Tear down and rebuild Docker stacks while reapplying Prisma state.

Community
Advanced