evals

You can run inline model evaluation during training by integrating custom evaluators into your SL or RL training loop. These evaluators execute at configurable intervals using the eval_every parameter, computing metrics across problems to monitor progress and detect drift automatically.

Inspect AI integration provides standard benchmarks and traceable evaluation results for language model training. It connects to your SL and RL workflows through utility functions, allowing you to measure model performance against established benchmarks during the fine-tuning process.

Yes, you can define custom evaluators as async functions that accept a SamplingClient or training context. This allows you to implement specialized evaluation patterns, including sampling-based and NLL-based evaluation, to compute custom metrics across your specific training problems.

Yes, inline evaluators are designed for both supervised learning (SL) and reinforcement learning (RL) training workflows. You can use the same evaluator_builders and eval_every configuration to run real-time evaluations and report metrics automatically across both training paradigms.

The best way to monitor for model drift during fine-tuning is to use inline evaluators at configurable intervals. By integrating custom metrics computation and Inspect AI benchmarks directly into your training loop, you can detect performance changes and validate model behavior in real-time.