skills.rest
MCPJam
MCPJamOFFICIAL
Explore all skills by this author →
Website𝕏@mcpjamsGitHub

mcpjam-eval-import

Convert promptfoo, pytest, Jest, and CSV test corpora into validated MCPJam eval suite YAML files.

2.2k|272|Updated May 23, 2025
Search Tags:#Jest#promptfoo#PyTest#test conversion#YAML validation#mcpjam#eval migration
One-click install
npx skills add https://github.com/MCPJam/inspector --skill mcpjam-eval-import
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: mcpjam-eval-import
Source: https://github.com/MCPJam/inspector/tree/main/skills/mcpjam-eval-import
Command: npx skills add https://github.com/MCPJam/inspector --skill mcpjam-eval-import

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Teams that already maintain prompt tests in promptfoo, pytest, Jest, or CSV exports must manually rewrite them to adopt MCPJam eval suites, risking silent semantic drift and lost test coverage during migration.

Core Features & Use Cases

  • Format-specific conversion recipes: Maps promptfoo YAML, pytest, Jest/Vitest, and CSV test cases onto MCPJam suite cases using cited structural rules (PF-, PY-, JS-, CSV-).
  • Honest import status tracking: Stamps every converted case with an exact, approximated, unsupported, or unresolved status, disabling non-exact cases until a human reviews them.
  • Offline validation loop: Runs mcpjam cloud eval validate until exit 0, with provenance hashes linking the suite file to its source artifact and mapping report.
  • Use Case: A repo with 40 promptfoo billing tests is converted into .mcpjam/evals/billing.yaml, where verbatim prompt and assertion mappings become enabled exact cases while LLM-rubric tests are parked as disabled approximated cases for review.

Quick Start

Ask the AI to import the promptfoo tests in this repository into an MCPJam eval suite and validate the resulting suite file.

Frequently Asked Questions about mcpjam-eval-import

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I import promptfoo tests into MCPJam?▼

Read the promptfoo config as text, map each tests[] item to one MCPJam case using the promptfoo recipe rules PF-1 through PF-5, write the suite to .mcpjam/evals/*.yaml, then run npx mcpjam cloud eval validate until it exits 0.

How to migrate pytest or Jest tests to an MCPJam eval suite?▼

Follow the pytest or Jest recipes: one literal test or parametrized row becomes one case, literal prompts are copied verbatim, and plain assertions map to responseContains or responseMatches. Never execute the source tests or fixtures during conversion.

What does the exact import status mean in MCPJam evals?▼

An exact status claims the conversion preserved structure under cited rules: one-to-one case mapping, verbatim prompts, identical assertion semantics, and nothing lost or invented. Only exact cases run without human review, and the claim is never independently verified by MCPJam.

Can MCPJam import LLM rubric or JavaScript assertions?▼

Model-graded rubrics are carried into expectedOutput as approximated cases because a judge grades differently from the source grader. Executable JavaScript or Python assertions with no deterministic predicate are marked unsupported and disabled.

Why does mcpjam cloud eval validate exit with code 1 or 2?▼

Exit 1 means the file parsed but violates the suite contract, with every finding listed by pointer; fix all findings and re-run. Exit 2 means nothing was judged due to an unreadable path, malformed YAML, or exceeding the 1 MiB cap.

Does the offline validator check tool names against a real server?▼

No. Offline validation only checks the file against the contract. Add --project <id-or-name> to resolve deterministic toolCall steps against a project's live tool inventory, which requires authentication and network access.

Related Skills

Explore More Software Engineering Skills→
promptfoopromptfoo
24.7k

promptfoo-evals

Write, run, and QA promptfoo eval suites with assertions and model-graded rubrics.

Official
Intermediate
eclipse-langiumeclipse-langium
28

lai-gen-evals

Generate comprehensive .eval.ts evaluation suites for Langium DSL projects.

Official
Advanced
elasticelastic
21.2k

evals-create-suite

Scaffold LLM evaluation suite packages with Playwright configuration files.

Official
Intermediate
elasticelastic
21.2k

review-data-source

Validate data source configurations for compliance and quality.

Official
Advanced