run-mcpjam-evals

Call run_eval_suite with the project to launch asynchronously and receive a runId, then poll get_eval_run until the status is completed, failed, or cancelled. Read the verdict from the run's decisionSummary before drilling into iterations.

Start with get_eval_run's decisionSummary and verdictSource to identify what decided the outcome. Then use list_eval_run_iterations to find failing cases and get_eval_run_steps with the runId and iterationId to see the exact failed step and its evidence.

run_eval_suite and request_eval_run_judge spend the organization's model budget and are marked COSTS MONEY. Confirm the suite, case count, and iteration multiplier before the first call, and pass an idempotencyKey if a launch might be retried.

Use compare_eval_run with a baseRunId or baseCommitSha. It classifies each case as regressed, fixed, new_case, removed_case, changed, unchanged_passed, or unchanged_failed, which separates new breakage from pre-existing failures.

list_eval_run_iterations is paginated, so a single page does not represent the whole run. Pass the returned nextCursor back as cursor until it is absent before computing any pass rate.

Use this skill to run, investigate, or compare evals that already exist in an MCPJam project. To author eval files in a repo, use create-mcp-eval for SDK tests or mcpjam-eval-import to convert an existing corpus.