keyword-based-llm-eval

You can evaluate LLM structured outputs without embeddings by using keyword matching and F1-based scoring. This lightweight approach compares generated text against expected keywords to calculate recall, precision, and F1 metrics quickly.

Keyword-based evaluation for LLM extractions is a method that quantifies output quality by checking for expected keywords. It computes recall, precision, and F1 scores per case, enabling fast tracking of prompt quality over time without semantic similarity dependencies.

To calculate precision and recall for LLM generated summaries, provide a YAML dataset describing cases and expected keywords. The evaluator matches keywords against the LLM output and aggregates these matches into precision, recall, and F1 metric reports.

Yes, keyword matching works for evaluating LLM outputs when exact wording varies. By configuring per-case keywords rather than strict exact-match strings, the evaluator accommodates phrasing differences while still providing quantitative F1 scoring for structured outputs.

A limitation of F1 keyword scoring for LLM evaluation is that it focuses strictly on keyword presence rather than semantic meaning. It is best suited for structured outputs like cards and extractions, and may miss nuanced semantic accuracy that embedding-based evaluations capture.

Yes, you can use YAML datasets to configure LLM evaluation metrics. The evaluator includes a YAML dataset loader that reads your cases and expected keywords, then processes them through a keyword matcher to aggregate recall, precision, and F1 calculations.