keyword-based-llm-eval

Name: keyword-based-llm-eval
Availability: InStock
Author: shimo4228

Evaluate structured LLM outputs with keyword matching and F1 scoring.

Updated Aug 23, 2026

Search Tags:#recall #llm #evaluation #precision #structured-output #keyword #f1

Ready to run

npx skills add https://github.com/shimo4228/claude-code-learned-skills --skill keyword-based-llm-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: keyword-based-llm-eval
Source: https://github.com/shimo4228/claude-code-learned-skills/tree/main/skills/keyword-based-llm-eval
Command: npx skills add https://github.com/shimo4228/claude-code-learned-skills --skill keyword-based-llm-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluates LLM-generated structured outputs against expected results using keyword matching and F1 metrics, offering a fast, low-dependency alternative to semantic similarity approaches.

Core Features & Use Cases

Lightweight keyword-based evaluation for structured outputs such as cards, summaries, and extractions.
Per-case keyword configurations and scoring to compute recall, precision, and F1.
Aggregated reports and comparisons to track prompt quality over time.

Quick Start

Provide a YAML dataset describing cases and expected keywords, then run the evaluator to obtain recall, precision, and F1.

Frequently Asked Questions about keyword-based-llm-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate LLM structured outputs without using embeddings?▼

You can evaluate LLM structured outputs without embeddings by using keyword matching and F1-based scoring. This lightweight approach compares generated text against expected keywords to calculate recall, precision, and F1 metrics quickly.

What is keyword-based evaluation for LLM extractions?▼

Keyword-based evaluation for LLM extractions is a method that quantifies output quality by checking for expected keywords. It computes recall, precision, and F1 scores per case, enabling fast tracking of prompt quality over time without semantic similarity dependencies.

How do I calculate precision and recall for LLM generated summaries?▼

To calculate precision and recall for LLM generated summaries, provide a YAML dataset describing cases and expected keywords. The evaluator matches keywords against the LLM output and aggregates these matches into precision, recall, and F1 metric reports.

Does keyword matching work for evaluating LLM outputs when exact wording varies?▼

Yes, keyword matching works for evaluating LLM outputs when exact wording varies. By configuring per-case keywords rather than strict exact-match strings, the evaluator accommodates phrasing differences while still providing quantitative F1 scoring for structured outputs.

What are the limitations of using F1 keyword scoring for LLM evaluation?▼

A limitation of F1 keyword scoring for LLM evaluation is that it focuses strictly on keyword presence rather than semantic meaning. It is best suited for structured outputs like cards and extractions, and may miss nuanced semantic accuracy that embedding-based evaluations capture.

Can I use YAML datasets to configure LLM evaluation metrics?▼

Yes, you can use YAML datasets to configure LLM evaluation metrics. The evaluator includes a YAML dataset loader that reads your cases and expected keywords, then processes them through a keyword matcher to aggregate recall, precision, and F1 calculations.