troubleshooting-network-operator

Diagnose NVIDIA Network Operator failures in Kubernetes clusters using must-gather logs and kubectl checks.

7|4|Updated Oct 24, 2024
One-click install
npx skills add https://github.com/Mellanox/netop-tools --skill troubleshooting-network-operator-mellanox
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: troubleshooting-network-operator
Source: https://github.com/Mellanox/netop-tools/tree/main/skills/troubleshooting-network-operator
Command: npx skills add https://github.com/Mellanox/netop-tools --skill troubleshooting-network-operator-mellanox

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When RDMA networking breaks in a Kubernetes cluster running the NVIDIA Network Operator, finding the root cause is slow: pods lose network attachments, SR-IOV VFs disappear, IPAM pools exhaust, and operator pods crash without obvious signals. This Skill provides a systematic diagnostic flow that maps symptoms to the exact commands and log files needed to isolate the fault. ## Core Features & Use Cases - Guided Diagnostic Flow: Step-by-step checks starting from must-gather collection, through operator health verification, to targeted checks for network attachment, SR-IOV, IPAM, and connectivity issues. - Symptom-to-Cause Decision Tree: A lookup table mapping common symptoms (CrashLoopBackOff, stuck Pending pods, missing VFs, IP allocation failures) to the specific check command and likely root cause. - Stuck Resource Cleanup: Commands to find orphaned finalizers, force-delete terminating namespaces, and remove evicted pods. - Use Case: A cluster admin notices application pods are stuck Pending with no RDMA network. Following the Skill, they run the SR-IOV sync checks, discover the PCI BDF in NETOP_NETLIST is wrong, fix the config, and verify VFs appear. ## Quick Start Ask the assistant to troubleshoot why pods in your Kubernetes cluster are not getting RDMA network from the NVIDIA Network Operator.

Frequently Asked Questions about troubleshooting-network-operator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot NVIDIA Network Operator issues in Kubernetes?

Start by running the must-gather script to collect full diagnostics, then verify all operator pods are Running and NicClusterPolicy shows state ready. Use the symptom decision tree to pick targeted checks for SR-IOV, IPAM, or network attachment problems.

Why are SR-IOV Virtual Functions not showing up on my node?

Missing VFs usually mean the SR-IOV sync is still in progress or the PCI BDF in NETOP_NETLIST is wrong. Check sync status with checksriovstate.sh, wait for sync with syncsriov.sh, and verify actual VF counts with getnumvfs.sh.

How do I fix IP allocation failures with Network Operator IPAM?

IP allocation failures typically indicate an exhausted pool, a wrong IPAM_TYPE setting, or a pool that was never applied. Inspect node IPAM annotations with checkipam.sh, review pool definitions with getippool.sh, and list allocated IPs with getallocatedip.sh.

Why is my pod stuck Pending with no RDMA network attached?

A Pending pod usually means the requested network resource name is unavailable or the NetworkAttachmentDefinition is missing from the pod's namespace. Run kubectl describe pod, then verify attachment definitions with get-network-attach-defs.sh and pod annotations with getnetworkstatus.sh.

How do I delete namespaces stuck in Terminating state in Kubernetes?

Stuck namespaces are usually blocked by orphaned finalizers. Find them with getfinalizers.sh, patch the resource to remove the finalizer, and use delstucknamespace.sh to force-delete the terminating namespace.