sop-by-action-eval

Automate Vision-Language Model evaluation for SOP monitoring by submitting by-action inference jobs to the BP evaluation-ms HTTP API. This Skill polls for terminal results and collects accuracy metrics, eliminating manual status tracking and output discovery.

By-action inference evaluation measures per-action accuracy for completed Vision-Language Model training jobs. It submits jobs to the evaluation-ms service, polls until completion, and returns a structured JSON envelope containing overall accuracy, evaluation paths, logs, and inference artifacts.

Yes, a reachable evaluation-ms HTTP service is required to run per-action VLM evaluation. You also need valid training and validation dataset identifiers, a host results mapping, and PyYAML installed for configuration parsing.

Yes, you can override evaluation parameters when starting a SOP action accuracy job. Provide natural-language parameter overrides alongside an inputs YAML file, and the Skill will apply these configuration overrides before submitting the by-action evaluation job.

By-action evaluation jobs fail to return accuracy metrics when the evaluation-ms service is unreachable, dataset identifiers are invalid, or polling times out. The Skill surfaces these failures without attempting unsafe automatic fixes.

The best way to get inference artifacts for root-cause analysis is to run by-action evaluation against a registered validation dataset. The Skill outputs a JSON result envelope containing logs, evaluation paths, and inference artifacts for downstream investigation.