online-evaluations

Use the langwatch CLI to create a monitor with a valid evaluator type, the correct level (trace or thread), and ON_MESSAGE execution mode. First inspect existing monitors with langwatch monitor list, then verify the created monitor appears with the intended configuration.

An online evaluation scores live traces or threads asynchronously for quality monitoring. A guardrail runs synchronously and can block or replace unsafe requests and responses, such as jailbreak detection or PII filtering.

Use trace level for per-interaction quality scoring of individual requests. Use thread level for multi-message conversation outcomes, and configure an appropriate idle timeout in the platform when needed.

No. Batch experiments, prompt comparisons, benchmarks, and CI quality gates belong to the separate experiments skill. This skill only handles production traffic monitoring and synchronous guardrails.

A validation_error names the incorrect field and an expected list of valid values. Correct that exact field using a value from the list and retry once; take evaluator types from langwatch evaluator types rather than guessing.

Work within the stated limits by creating only the allowed number of meaningful resources. If a limit is hit, summarize what was accomplished and point to subscription settings for upgrading, without deleting existing resources.