What problem does it solve?
Testing deployed ACE OCS chatbots for Connect opportunities previously required manual interaction, ad-hoc note-taking, and inconsistent validation of response quality, leading to missed issues like timeouts, missing citations, or broken RAG for opp-specific content before launch or during production monitoring.
Core Features & Use Cases
- Multi-Mode Testing: Supports three testing modes tailored to different lifecycle stages: a 3-prompt quick smoke test for post-setup Phase 5 validation, a full deep suite with opp-specific prompts for pre-launch QA, and a recurring monitor suite for Phase 6 production health checks.
- Built-in Structural Validation: Automatically runs deterministic checks on every chatbot response to flag timeouts, error responses, and missing citations, with a 3-prompt circuit breaker to halt testing if the OCS instance is unhealthy.
- Durable Transcript Capture: Writes machine-readable, structured transcripts to Google Drive that serve as direct input for the paired ocs-chatbot-eval skill, with support for resuming partial captures if the suite hits wall-clock limits or is interrupted.
- Use Case: A product manager running a pre-launch check for a new focus-group opportunity can use deep mode to test both shared Connect workflow questions and opp-specific guidance questions, catching RAG misconfigurations before the chatbot goes live to frontline workers.
Quick Start
Use the ocs-chatbot-qa skill to run a full deep QA suite against your deployed ACE OCS chatbot for the 'malaria-itn-fgd' opportunity and capture a structured transcript for grading.
