creating-online-evaluations

Start from a real failure mode found in production traces, propose candidate evals for the user to pick, then create each one disabled with the llma-evaluation-create tool. Verify the condition scope with a SQL volume check before enabling it with llma-evaluation-update.

Use hog for structural, rule-based criteria like JSON parsing, length, regex, or tool-call shape since it is deterministic and needs no provider key. Use llm_judge only for subjective criteria like tone or on-topic checks, which cost an LLM call per run and require a usable provider key.

A generation target scores each matching $ai_generation immediately after ingestion. A trace target scores the whole trace once after a settle window, and a session target scores an entire $ai_session_id session after it goes quiet. Sentiment evals only support the generation target.

Common causes are conditions that match no events, no $ai_generation events in the project at all, or a session target where producers never set $ai_session_id. Verify scope by running a SQL count over the last 7 days mirroring each condition property before enabling.

Yes, an llm_judge eval needs a valid provider and model plus a usable provider key in the ok state. You can pin a specific key via provider_key_id or leave it null only when the team's active key is usable and matches the model's provider.

Propose a sentiment eval as the floor, since it labels user messages positive, neutral, or negative with no judge cost. If the project has no $ai_generation events at all, no eval can fire, so set up LLM analytics instrumentation first.