evaluator

You can automate agent evaluation metrics by using the NeMo Platform SDK to orchestrate workspace creation, dataset preparation, and metric execution without manual CLI intervention. This streamlines the entire metric lifecycle from start to finish.

Asynchronous metric job monitoring allows you to trigger long-running background evaluation jobs and poll their status using the nemo_api evaluation module. This enables you to retrieve results later without blocking your development environment.

Yes, synchronous execution supports immediate inline metric evaluation for quick checks. This allows you to perform rapid string-check evaluations on new agent versions and get instant feedback without waiting for background processing.

Yes, integration with the nemo_api evaluation module is required to handle metric operations and job status polling. This dependency allows the automation process to manage both synchronous and asynchronous evaluation jobs effectively.

The primary limitation is that metric lifecycle management is tightly coupled to the NeMo Platform environment. Users must operate within this specific ecosystem to successfully execute synchronous and asynchronous evaluation jobs.