checkpoints

To save training state for resumption, use the save_state function, which persists both model weights and optimizer state. This allows you to reliably restart interrupted training workflows from the exact checkpoint without losing progress.

The best way to export sampler weights for evaluation is using save_weights_for_sampler. Alternatively, you can use save_weights_and_get_sampling_client to export weights and immediately obtain a ready-to-use sampling client for deployment.

Yes, you can manage checkpoint lifecycle and provenance through a built-in REST API and CLI. These interfaces allow you to list available checkpoints, publish them, set TTL for automatic cleanup, and delete records as needed.

CheckpointRecord provides serialization and deserialization for robust tracking and reproducibility of training checkpoints. It structures the saved state and sampler data, ensuring consistent records can be reliably restored or audited.

You should save only sampler weights with save_weights_for_sampler when you need fast evaluation or deployment sampling. Save full state with save_state only when you need to resume training and require optimizer state persistence.

Yes, checkpointing handles both model weights and optimizer state. The save_state function captures the complete training state, ensuring that resumption preserves all optimizer momentum and parameters alongside the model weights.