llm-evaluation

LLM evaluation lets you run batch evaluations against multiple models simultaneously, tracking token usage and costs. Set up API keys in SETUP.md, then use call-llm-batch.py to execute parallel model calls and generate comparable performance metrics across providers.

Yes. The skill provides batch-processing pipelines with incremental result saves and parallel workers. Use generate-questions.py and generate-answers.py to orchestrate multi-step workflows, then evaluate-answers.py to score outputs while tracking token usage and computing per-model costs.

Build end-to-end evaluation pipelines using the skill's flexible prompt and scoring mechanisms. Transcribe or prepare input data, generate questions and answers with configurable LLM calls, then apply LLM-as-judge scoring. Results include token counts and cost breakdowns for each model variant.

Yes. The skill supports optional prompt caching for both OpenAI and Anthropic models. Enable caching in your pipeline configuration to lower token consumption and costs during repeated evaluations with the same prompts.

Both. Choose your workflow: single LLM calls via call-llm.py for quick tests, batch processing with call-llm-batch.py for larger runs, or the complete evaluation pipeline for end-to-end benchmarking. All share the same token-tracking and cost-analysis foundation.

The pipeline requires JSON metadata for prompts, scoring rules, and execution configuration. Provide input datasets with questions and reference answers; the skill handles token-usage tracking, result aggregation, and cost computation automatically during batch runs.