operator-eval

Automate operator-only evaluation loops over local runtime artifacts.

135|36|Updated Mar 21, 2026

Search Tags:#workflow #evaluation #provenance #regression #local-runtime #operator-eval

Ready to run

npx skills add https://github.com/JetXu-LLM/DocMason --skill operator-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: operator-eval
Source: https://github.com/JetXu-LLM/DocMason/tree/main/skills/operator/operator-eval
Command: npx skills add https://github.com/JetXu-LLM/DocMason --skill operator-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

The hidden operator-only evaluation loop over local runtime/eval artifacts can be difficult to run safely and reproducibly within standard user workflows.

Core Features & Use Cases

Automates local evaluation tasks by reading, validating, and executing a single operator request from the local surface.
Supports regression review, replayable evaluation runs, candidate promotion, and baseline freezing while preserving provenance.
Ensures isolation by keeping evaluation artifacts under runtime/eval and separating runtime activity logs from normal user data.

Quick Start

Ask the agent to run the operator-eval workflow for the current evaluation request and return results to the main agent.

Frequently Asked Questions about operator-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate local evaluation loops for regression review safely?▼

Automate local evaluation loops by reading, validating, and executing a single operator request from the local surface to safely run regression reviews while keeping artifacts isolated from normal user flows.

What is an operator-only evaluation workflow for local runtime artifacts?▼

An operator-only evaluation workflow is a hidden, isolated loop that processes local runtime artifacts to perform replayable evaluation runs, candidate promotion, and baseline freezing without exposing these operations to normal users.

How do I freeze evaluation baselines while preserving provenance locally?▼

Freeze evaluation baselines by executing operator requests over local runtime artifacts, which preserves provenance and separates runtime activity logs from standard user data during the process.

Can I run replayable evaluation runs without affecting normal user data?▼

Yes, you can run replayable evaluation runs by keeping all evaluation artifacts under runtime/eval directories, ensuring isolation from normal user data and workflows.

What are the limitations of running operator-only eval workflows locally?▼

Operator-only eval workflows require local file access, shell execution capabilities, and structured JSON output inspection, keeping all actions strictly out of standard user flows and confined to local runtime environments.

Does operator-eval work with structured JSON output for candidate promotion?▼

Yes, operator-eval inspects structured JSON output from local runtime artifacts to automate candidate promotion, returning operator-facing results and artifacts directly to the main agent.