skills.rest
biocypher
biocypherOFFICIAL
Explore all skills by this author →
WebsiteGitHub

karenina-task-eval

Evaluate pre-recorded LLM outputs against templates and rubrics using karenina's TaskEval API.

13|3|Updated Jun 27, 2025
Search Tags:#llm evaluation#rubric#agent traces#judge model#karenina#taskeval#answer template
One-click install
npx skills add https://github.com/biocypher/karenina --skill karenina-task-eval-biocypher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: karenina-task-eval
Source: https://github.com/biocypher/karenina/tree/main/skills/karenina-task-eval
Command: npx skills add https://github.com/biocypher/karenina --skill karenina-task-eval-biocypher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and assets (resource) components.

What problem does it solve? You have existing model responses, chatbot logs, or agent traces and need to assess them against structured criteria without re-running an LLM. This Skill guides you through karenina's TaskEval API to run judge-only verification on pre-collected outputs. ## Core Features & Use Cases - Pre-Recorded Output Evaluation: Log plain text or structured conversation traces and evaluate them with the same template/rubric engine as Benchmark mode, skipping answer generation. - Template and Rubric Verification: Attach BaseAnswer templates with VerifiedField definitions and optional LLM rubric traits for qualitative assessment. - Multi-Step Evaluation: Score outputs globally or per named step, with configurable merge strategies for combining logs before judging. - Use Case: You collected 200 chatbot responses from a production deployment. Log each response into a TaskEval instance, attach a template checking factual claims and a rubric for evidence citation, then run a judge-only pass with Claude to score every response. ## Quick Start Ask the AI to evaluate a pre-recorded model response with karenina TaskEval by logging the output, attaching an answer template, and running evaluate with parsing_only set to true.

Frequently Asked Questions about karenina-task-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate pre-recorded LLM outputs with karenina TaskEval?▼

Create a TaskEval instance, log outputs with .log() or .log_trace(), attach a BaseAnswer template via add_template(), then call evaluate() with a VerificationConfig that sets parsing_only=True and a parsing judge model.

How to evaluate LLM outputs without running an answering model?▼

Set parsing_only=True in VerificationConfig so the pipeline skips answer generation and runs only the judge pass. TaskEval auto-detects the evaluation mode from attached templates and rubrics.

Why does TaskEval fail with an answering model error?▼

The VerificationConfig is missing parsing_only=True, so the pipeline tries to invoke a non-existent answering LLM. Always set parsing_only=True when evaluating pre-recorded outputs with TaskEval.

Can TaskEval score multi-step agent traces separately?▼

Yes. Log outputs with a step_id, attach step-scoped questions or rubrics, and call evaluate(). The global pass runs first, then every step with data is evaluated automatically; steps-only evaluation also works.

Why does my TaskEval template only parse the last logged message?▼

Template parsing reads only the final AI message by default because use_full_trace_for_template is False. Put all text in one .log() call or set use_full_trace_for_template=True in VerificationConfig.

Related Skills

Explore More Data & Analytics Skills→
langwatchlangwatch
3.5k

datasets

Generate realistic synthetic evaluation datasets and upload them to LangWatch.

Official
Advanced
Yuan-ZzzzYuan-Zzzz

unity-perception

Analyze Unity scenes and export structured summaries and dependency reports.

Community
Advanced
Miosa-osaMiosa-osa
226

judge-prompt

Design binary pass/fail LLM-as-Judge evaluation prompts with scoring instructions and guardrails.

Official
Advanced
diegosouzapwdiegosouzapw
59.5k

cli-eval

Create and run LLM evaluation suites, scorecards, and model comparisons from the OmniRoute CLI.

Community
Intermediate