skills.rest
Mellanox
MellanoxOFFICIAL
Explore all skills by this author →
WebsiteGitHub

managing-netop-devices

Manages SR-IOV virtual functions, PCI devices, link speed, and RDMA tests on Kubernetes nodes.

7|4|Updated Oct 24, 2024
Search Tags:#kubernetes#node management#virtual functions#rdma#nvidia network operator#sr-iov#link speed
One-click install
npx skills add https://github.com/Mellanox/netop-tools --skill managing-netop-devices-mellanox
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: managing-netop-devices
Source: https://github.com/Mellanox/netop-tools/tree/main/skills/managing-netop-devices
Command: npx skills add https://github.com/Mellanox/netop-tools --skill managing-netop-devices-mellanox

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Operating NVIDIA Network Operator clusters requires repetitive low-level device tasks—configuring SR-IOV virtual functions, forcing link speeds, labeling nodes, and validating RDMA connectivity—which are error-prone when done manually across many worker nodes. ## Core Features & Use Cases - SR-IOV and PCI Management: Set VF counts on PCI devices, query vendor IDs, and inspect network device details on worker nodes. - RDMA Testing: Run RoCE, InfiniBand, and GPU Direct RDMA bandwidth tests between server and client pods, plus CUDA performance tests. - Node Operations: Label workers and control planes, manage taints, cordon/uncordon nodes, and deploy test application pods. - Use Case: After installing Network Operator on a DGX cluster, use this Skill to set 8 VFs on each ConnectX device, verify RDMA capability, and run an ib_send_bw test between two pods to confirm fabric performance. ## Quick Start Ask the agent to set 8 SR-IOV virtual functions on PCI device 0000:08:00.0 and then verify RDMA capability on the worker node.

Frequently Asked Questions about managing-netop-devices

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set SR-IOV virtual functions on NVIDIA network devices?▼

Run the setvfs.sh script on the worker node with the desired VF count and PCI bus addresses, for example ./setvfs.sh 8 0000:08:00.0. You can verify the current VF count afterward with ./ops/getnumvfs.sh.

How to test RDMA bandwidth between Kubernetes pods?▼

Run rocesrv.sh on the server pod and roceclnt.sh on the client pod for RoCE testing, or use rdmasrv.sh and rdmaclnt.sh for InfiniBand ib_send_bw tests. GPU Direct RDMA tests use gdrsrv.sh and gdrclt.sh.

How do I force a specific link speed on a Mellanox interface?▼

Use ./ops/force_link_speed.sh with the device name, speed, and lane width, such as ./ops/force_link_speed.sh mlx5_0 400G 4X. This disables auto-negotiation, and the XVAL must match the chosen speed.

Does disabling PCI ACS work on virtualized nodes?▼

The disable_acs.sh script for peer-to-peer RDMA is intended for bare metal nodes only. Virtualized environments handle PCI passthrough differently, so ACS disabling may not apply or be permitted.

How do I cordon and uncordon worker nodes in the cluster?▼

Source the cordon.sh script with source ${NETOP_ROOT_DIR}/ops/cordon.sh, then call the cordon or uncordon functions to act on all worker nodes. Individual taints can be viewed with gettaints.sh and removed with rmtaints.sh.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
14

k8s-launch-kit-deploy

Apply generated NVIDIA networking manifests to a Kubernetes cluster in dependency order.

Official
Advanced
NVIDIANVIDIA
14

k8s-launch-kit-troubleshoot

Diagnose NVIDIA Network Operator and Kubernetes networking failures from sosreport files.

Official
Advanced
Project-HAMiProject-HAMi
4.3k

k8s_gpu_pod_troubleshooter

Diagnose GPU pod scheduling and allocation issues in Kubernetes clusters with HAMi.

Official
Advanced
CoWork-OSCoWork-OS
420

developer-growth-analysis

Analyze Claude Code chat history and deliver a growth report to Slack DMs.

Official
Advanced