skills.rest
MRiabov
MRiabovCOMMUNITY
Explore all skills by this author →
GitHub

eval-creation-workflow

Create or repair Problemologist eval seeds with deterministic fields and run_evals verification.

2|Updated Jan 31, 2026
Search Tags:#validation#dataset#llm-evaluation#eval-seed#seed-workflow#handover-contracts
One-click install
npx skills add https://github.com/MRiabov/Problemologist-AI --skill eval-creation-workflow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-creation-workflow
Source: https://github.com/MRiabov/Problemologist-AI/tree/main/.agents/skills/eval-creation-workflow
Command: npx skills add https://github.com/MRiabov/Problemologist-AI --skill eval-creation-workflow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Create or repair Problemologist eval seeds by adding role-based dataset rows plus stage-correct seeded workspace artifacts with exact deterministic fields, then verify them with minimal-scope runs through dataset/evals/run_evals.py. Use this when asked to add benchmark, engineer, or reviewer evals, or when a role-based eval dataset looks structurally invalid.

Core Features & Use Cases

  • Seeds role-based dataset rows and stage-aligned workspace artifacts with exact deterministic fields to enable reliable evaluation, auditing, and reproducibility.
  • Supports adding benchmark, engineer, and reviewer evals, and helps fix structurally invalid eval datasets through guided references and deterministic checks.
  • Validates seeds with minimal-scope runs via dataset/evals/run_evals.py to quickly detect schema or contract mismatches before full evaluation.

Quick Start

Seed a new eval for a target role by populating role-based dataset rows and seeded workspace artifacts with exact deterministic fields, then verify the seed with a minimal-scope run of dataset/evals/run_evals.py.

Frequently Asked Questions about eval-creation-workflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create reproducible LLM evaluation datasets with deterministic fields?▼

To create reproducible LLM evaluation datasets, seed role-based dataset rows and stage-aligned workspace artifacts using exact deterministic fields. This ensures reliable auditing and consistent evaluation runs.

How do I fix a structurally invalid eval dataset?▼

Repair structurally invalid eval datasets by re-injecting role-based dataset rows with exact deterministic fields. Guided references and deterministic checks help restore schema and contract compliance.

How do I validate new eval seeds before running a full evaluation?▼

Validate new eval seeds by executing minimal-scope runs via dataset/evals/run_evals.py. This quickly detects schema or contract mismatches before committing to a full evaluation process.

Can I add benchmark, engineer, and reviewer evals to the same dataset?▼

Yes, you can add benchmark, engineer, and reviewer evals by injecting role-based dataset rows into the seed. Each role receives stage-correct seeded workspace artifacts for reliable evaluation.

What are handover-contracts in LLM evaluation workflows?▼

Handover-contracts are guidelines that ensure deterministic field compliance during eval seed creation and repair. Adhering to active handoff-contracts guarantees reproducibility and structural validity across evaluation stages.

Related Skills

Explore More Data & Analytics Skills→
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
yonatangrossyonatangross
3

golden-dataset-validation

Validate golden datasets for schema, duplicates, and referential integrity.

Community
Advanced
sinfinysinfiny
5

autonomous-webserial-loop

Automate iterative planning, drafting, and revision of mystery webserial chapters.

Community
Advanced
hungv47hungv47
14

evaluate-campaign

Score post-launch campaigns across paid, organic, and email channels with verdicts and unit economics.

Community
Advanced