karenina-qa

Create a Benchmark object, add Question objects with question text and raw_answer ground truth, attach answer templates (auto-generated or manual), then call run_verification with a VerificationConfig specifying answering and parsing models.

Use extract_questions_from_file with the file path, question column, and answer column names. It supports CSV, TSV, and Excel, plus optional keyword and metadata columns, returning Question objects for add_questions.

No. Benchmark.save() persists only questions, templates, and rubrics in JSON-LD. Verification results stay in memory and must be exported separately via ResultsStore with add() and export_to_file().

The default evaluation_mode is template_only, which skips attached rubrics. Set evaluation_mode to template_and_rubric or rubric_only explicitly in the VerificationConfig to run rubric evaluation.

The provider API key must be present in the process environment, such as ANTHROPIC_API_KEY exported in the shell. The repo-root .env is not auto-loaded, so source it explicitly or pass the key via ModelConfig.

Question.id is the MD5 hash of the question text, so identical text produces the same ID. Adding a duplicate silently overwrites the previous entry including its template; change the text to get a distinct ID.