nemo-automodel-model-onboarding

Guides onboarding of new LLM, MoE, and VLM architectures into NeMo AutoModel.

960|316|Updated May 21, 2025
One-click install
npx skills add https://github.com/NVIDIA-NeMo/Automodel --skill nemo-automodel-model-onboarding-nvidia-nemo
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: nemo-automodel-model-onboarding
Source: https://github.com/NVIDIA-NeMo/Automodel/tree/main/skills/nemo-automodel-model-onboarding
Command: npx skills add https://github.com/NVIDIA-NeMo/Automodel --skill nemo-automodel-model-onboarding-nvidia-nemo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Adding support for a new Hugging Face model architecture in NeMo AutoModel involves many error-prone steps: classifying the architecture, implementing model files, writing state-dict adapters, registering classes, and validating numerical parity. This Skill provides a structured five-phase workflow with checklists and reference patterns so nothing is missed. ## Core Features & Use Cases - Architecture Discovery: Classify dense LLM, MoE, or VLM models from config.json fields such as architectures, expert counts, and vision_config/text_config. - Implementation Patterns: Reference guides for dense LLMs, MoE state-dict adapters (router, expert, and shared-expert weight mapping), and VLM wrappers (vision tower, projector, language model). - Registration & Validation: Register classes in MODEL_ARCH_MAPPING, declare ModelCapabilities and weight-tying policies, and run unit, layer-equivalence, and end-to-end parity tests. - Use Case: When adding support for a new Hugging Face causal LM, follow the checklist to create components/models/<name>/, register the class, add a tiny-config test, and verify checkpoint round-trip before full training. ## Quick Start Ask the agent to onboard a new Hugging Face model architecture into NeMo AutoModel, for example by saying: add support for the new model in my config.json as a NeMo AutoModel architecture.

Frequently Asked Questions about nemo-automodel-model-onboarding

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new Hugging Face model architecture to NeMo AutoModel?

Fetch the model's config.json, classify it as dense LLM, MoE, or VLM, then create files under components/models/<name>/ following the pattern guides. Register the class in MODEL_ARCH_MAPPING in _transformers/registry.py, add an example YAML, and write tiny-config unit tests before loading full checkpoints.

How do I write a state-dict adapter for MoE expert weights?

Map router weights, routed and shared experts, and gate/up/down projections explicitly, preserving expert index order between HF per-expert tensors and NeMo stacked tensors. Add key-mapping tests and tiny-config numerical equivalence tests rather than relying only on from_pretrained().

When is a state_dict_adapter.py file not needed?

Omit the adapter when HF weight names and tensor layouts already match across supported backend and config variants, as with Llama, Qwen2, and Qwen3. Weight tying remains the model's own responsibility even without an adapter.

Does NeMo AutoModel support VLM onboarding with vision towers?

Yes, VLMs with vision_config, text_config, and a ForConditionalGeneration architecture are supported via a wrapper combining vision tower, multi-modal projector, and language model. Follow vlm-patterns.md and existing implementations like mistral4 or kimivl, and add tiny image-text tests.

What tests are required before using full checkpoints?

Add forward-shape smoke tests with tiny configs, state-dict round-trip tests, HF load/export and native save/reload checks, layer-equivalence tests for rewritten layers, and a short functional test verifying loss decreases. Then run state-dict, component, and end-to-end parity comparisons against Hugging Face.

When should I not use this model onboarding skill?

Do not use it for standalone training recipe YAML questions about optimizers, datasets, schedulers, or distributed strategy choices like FSDP2 or tensor parallelism. Those belong to the nemo-automodel-recipe-development skill unless explicitly tied to onboarding a new architecture.