ocs-chatbot-eval

Chatbot transcript grading can be automated using LLM-as-Judge rubrics to evaluate captured transcripts against standardized criteria. This process replaces manual quality checks with calibrated scoring, producing machine-readable verdicts and human-readable evaluation reports for consistent, auditable results.

LLM-as-Judge rubric calibration applies hard deduction rules and inflation guards to standardize chatbot evaluation scoring. Pre/post-cap scoring is calibrated against ground truth to ensure defensible, consistent grading across multiple evaluation runs without human bias.

Run a quick grading mode using a 3-prompt shallow smoke test on captured chatbot transcripts to validate critical functionality. This catches major failures in minutes and generates a gate brief to determine if the chatbot passes the quality gate before advancing to deployment.

Calibrated rubrics can be used in a monitor grading mode for recurring performance trend tracking of chatbot deployments. This ongoing evaluation applies the same standardized LLM-as-Judge criteria to monitor quality over time and integrate results into an aggregation workflow.

Automated chatbot transcript evaluation generates machine-readable verdict YAML, human-readable evaluation reports, and gate briefs. These outputs integrate directly with upstream orchestration gates and aggregation workflows to pass quality data into downstream systems.

Use deep mode for pre-launch validation evaluating chatbots against 5 calibrated dimensions, while quick mode handles rapid 3-prompt smoke tests to catch critical failures. Deep mode provides comprehensive assessment before releasing to users, whereas quick mode gates shallow quality checks.