tao-train-sparse4d

Run dataset_convert first to produce annotation pickles and anchor_init.npy, then launch train with mandatory data source overrides for data_root, anchor, and ann_file paths. The Skill routes training through AutoML by default unless automl_policy is set to off.

Sparse4D training requires the sparse4d dataset type in OVPKL format, converted from AICity source data using the tao_toolkit.data_services image with the annotations convert command. Conversion produces annotation pickles and the anchor_init.npy file needed by all downstream actions.

Sparse4D training requires a minimum of 2 GPUs with 40GB or more VRAM each, with A100 and 8 GPUs recommended. bf16 precision is required for practical training, and multi-GPU uses DDP with find_unused_parameters enabled.

The failure occurs when converted depth_map_path tuples point to camera directories instead of H5 files. Run the normalize_depth_paths.py script against the converted annotation directory after dataset_convert and before train, evaluate, or inference.

Yes, set automl_policy to off for the run, or use phrases like disable AutoML or plain training in your request. This applies only to that run and uses direct model training instead of routing through tao-run-automl.

Reduce dataset.num_frames or dataset.batch_size to lower memory usage during temporal training. The instance bank and multi-camera temporal reasoning are memory intensive, so smaller sequence lengths help fit within available VRAM.