karenina-template-authoring

Create and validate BaseAnswer templates for karenina LLM evaluation pipelines.

13|3|Updated Jun 27, 2025
One-click install
npx skills add https://github.com/biocypher/karenina --skill karenina-template-authoring-biocypher
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: karenina-template-authoring
Source: https://github.com/biocypher/karenina/tree/main/skills/karenina-template-authoring
Command: npx skills add https://github.com/biocypher/karenina --skill karenina-template-authoring-biocypher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? Writing answer templates for LLM-as-judge evaluation is error-prone: wrong base classes, mismatched ground truth types, vague field descriptions, and misused verification primitives all cause silent failures. This Skill guides the authoring of karenina answer templates (BaseAnswer subclasses with VerifiedField) so extracted values are verified correctly against ground truth. ## Core Features & Use Cases - Interactive six-step authoring workflow: from domain understanding through field identification, structure proposal, code generation, validation, and user confirmation. - Verification primitive guidance: covers BooleanMatch, ExactMatch, SetContainment, NumericGraded, LiteralMatch, trace primitives, and composition nodes (AllOf, AnyOf, AtLeastN) with a full primitive catalog reference. - Template validation script: runs structural checks on generated templates and reports errors for iterative fixing. - Use Case: You need to evaluate whether LLM responses correctly identify a drug's modality. The Skill walks you through defining a boolean VerifiedField with a judge-facing description, correct ground truth, and BooleanMatch, then validates the template before pipeline use. ## Quick Start Ask the assistant to create a karenina answer template for your evaluation question, describing the domain and what a correct answer looks like.

Frequently Asked Questions about karenina-template-authoring

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a karenina answer template for LLM evaluation?

Subclass BaseAnswer and define fields with VerifiedField, giving each a judge-facing description, a ground truth value, and a verification primitive such as BooleanMatch. Then run the validate_template.py script to check the template before using it in the pipeline.

Which verification primitive should I use for a template field?

Use BooleanMatch for yes/no judgments, ExactMatch for single-word or controlled-vocabulary strings, SetContainment for list extraction, and NumericTolerance or NumericGraded for numbers. BooleanMatch is the safest default when a fact can be phrased as a true/false question.

Why does my template fail verification even with correct answers?

Common causes are ground truth types that do not match the field annotation, using ExactMatch on free-text answers the judge phrases differently, or inheriting from pydantic BaseModel instead of BaseAnswer. BooleanMatch and ExactMatch coerce silently, so smoke-test verify() offline with known-good values.

Can I use ExactMatch for free-text extraction in karenina?

No. ExactMatch compares normalized strings for equality, and the judge LLM rarely reproduces the exact expected phrasing. For free-text content, convert the field to a boolean judgment with BooleanMatch and describe the expected content in the description.

How do I verify multiple extracted items in one field?

Use a list[str] field with SetContainment rather than a string field with ContainsAny. SetContainment supports exact, subset, superset, and overlap modes so you can control whether extra or missing items are allowed.