upgrading-network-operator

Upgrade NVIDIA Network Operator Helm releases across Kubernetes clusters with rollback support.

7|4|Updated Oct 24, 2024
One-click install
npx skills add https://github.com/Mellanox/netop-tools --skill upgrading-network-operator-mellanox
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: upgrading-network-operator
Source: https://github.com/Mellanox/netop-tools/tree/main/skills/upgrading-network-operator
Command: npx skills add https://github.com/Mellanox/netop-tools --skill upgrading-network-operator-mellanox

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Upgrading the NVIDIA Network Operator in a Kubernetes cluster involves coordinated steps—cordoning nodes, regenerating configs, applying CRDs, and running helm upgrade—where mistakes can disrupt RDMA networking for AI/ML workloads. This Skill guides the full upgrade workflow safely, including version compatibility checks and rollback. ## Core Features & Use Cases - Guided Upgrade Procedure: Step-by-step workflow covering pre-upgrade checks, dry-run config generation with CREATE_CONFIG_ONLY, and execution of upgrade-network-operator.sh. - Version-Specific Guidance: Documents behavioral differences across operator versions 24.7.0 through 26.1.0, including MAINTENANCE_OPERATOR_ENABLE defaults and API parameter changes. - Rollback and Troubleshooting: Provides rollback instructions and a failure table covering helm chart errors, CRD validation issues, stuck cordons, and finalizer blocks. - Use Case: A cluster administrator needs to move from Network Operator 25.7.0 to 26.1.0 on a DGX cluster; the Skill walks them through previewing config diffs, running the upgrade script, and verifying SR-IOV and IPAM state afterward. ## Quick Start Ask the AI to upgrade the NVIDIA Network Operator to version 26.1.0 and verify the cluster networking state afterward.

Frequently Asked Questions about upgrading-network-operator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I upgrade NVIDIA Network Operator in Kubernetes?

Set NETOP_VERSION to the target release, run a dry-run config generation with CREATE_CONFIG_ONLY=1 to review YAML diffs, then execute upgrade-network-operator.sh. The script cordons nodes, regenerates configs, applies CRDs, runs helm upgrade, and uncordons nodes.

How to roll back a failed Network Operator upgrade?

Restore the previous configuration by setting NETOP_VERSION back to the earlier release, sourcing global_ops.cfg, and re-running upgrade-network-operator.sh. This reapplies the prior NicClusterPolicy, CRDs, and network resources.

Which Network Operator versions are supported for upgrade?

Supported versions include 24.7.0, 24.10.0, 24.10.1, 25.1.0, 25.4.0, 25.7.0, 25.10.0, and 26.1.0 as the default. Version 26.1.0 requires Kubernetes 1.32 or later.

Why are nodes stuck cordoned after a Network Operator upgrade?

Nodes remain cordoned when the upgrade script fails before its uncordon step. Run source ops/cordon.sh followed by the uncordon command to restore scheduling on worker nodes.

What changed in Network Operator 26.1.0 for maintenance operator settings?

In 26.1.x, MAINTENANCE_OPERATOR_ENABLE defaults to true independently, while in 25.10.x it defaults to the NIC_CONFIG_ENABLE value. Review this flag when migrating between these versions.