setup-benchmark-inputs

To prepare MoE benchmarking artifacts, you can automate tokenizing DCLM corpus shards and converting HF checkpoints to DCP format. This generates minimal reproducible workspace artifacts for profiling and regression testing.

Setting up PithTrain benchmark artifacts involves orchestrating fetch, tokenize, import, and conversion steps via shell scripts. This idempotent process ensures required benchmark inputs exist at designated workspace paths without redundant reprocessing.

Yes, the MoE benchmark setup is idempotent and safe to re-run. It checks for existing tokenized corpus shards and converted HF checkpoints, automatically skipping steps when outputs already exist to prevent redundant processing.

Yes, the benchmark preparation supports multiple MoE models including deepseek-v2-lite and qwen3-30b-a3b. It generates model-specific outputs under the workspace, ensuring proper artifact configuration for each architecture.

Yes, huggingface_hub is required to fetch and convert HF checkpoints to DCP format. The setup integrates with a local Python environment to orchestrate the import and conversion needed for reproducible MoE profiling.

HF-DCP-formatted checkpoints are required for MoE regression testing to ensure compatibility with PithTrain workflows. Converting standard HF checkpoints to DCP format provides a standardized input layout for consistent profiling and regression validation.