One-click install
npx skills add https://github.com/konveyor/ai-rule-gen --skill eval-konveyor
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: eval
Source: https://github.com/konveyor/ai-rule-gen/tree/main/agents/eval
Command: npx skills add https://github.com/konveyor/ai-rule-gen --skill eval-konveyor

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you verify whether generated Konveyor migration rules are correct, complete, and aligned with the migration guide, instead of relying on guesswork after generation.

Core Features & Use Cases

  • Deterministic evaluation: Checks rule quality, app coverage, and overlap to show whether the rules work in practice.
  • LLM rule review: Compares each rule’s detection condition and migration message against the guide to catch wrong matches, vague guidance, and incorrect replacements.
  • Gap analysis: Finds migration patterns documented in the guide that have no corresponding rule and proposes concrete rule ideas.
  • Use cases: Validate a new ruleset before publishing, review regressions after edits, or audit a migration guide for missing coverage.

Quick Start

Ask the Eval skill to review your generated Konveyor rules against a migration guide and return a report of issues and missing migration patterns.

Frequently Asked Questions about eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate generated Konveyor migration rules against a migration guide?

To validate Konveyor migration rules, ingest the source migration guide and apply deterministic evaluation to check rule quality, sample-application coverage, and overlap, producing a structured findings report for precision issues.

What is ground-truth gap analysis for migration rules?

Ground-truth gap analysis finds migration patterns documented in the guide that have no corresponding rule, then proposes concrete rule ideas to ensure complete coverage of the migration requirements.

How do I review Konveyor rule detection conditions and migration messages?

Reviewing Konveyor rules involves comparing each rule's detection condition and migration message against the source guide to catch wrong matches, vague guidance, and incorrect replacements through LLM-based analysis.

Can I audit a migration guide for missing rule coverage before publishing a ruleset?

Yes, you can audit a migration guide for missing coverage by evaluating generated rules against the guide to find undocumented patterns and validate existing rules before publishing a new ruleset.

What metrics are checked during Konveyor rule evaluation?

Konveyor rule evaluation checks deterministic metrics including rule quality, sample-application coverage, and rule overlap to determine whether the generated migration rules work correctly in practice.

When should I evaluate migration rules for regressions after edits?

You should evaluate migration rules for regressions after edits by re-running the evaluation against the migration guide to verify that modified detection conditions and messages still align with the original requirements.