Processes chestnut images and generates a PDF report of findings.
npx skills add https://github.com/jjackson/ace --skill ocs-chatbot-eval
Please help me install this Agent Skill. Skill: ocs-chatbot-eval Source: https://github.com/jjackson/ace/tree/main/skills/ocs-chatbot-eval Command: npx skills add https://github.com/jjackson/ace --skill ocs-chatbot-eval
Quickly judges captured OCS chatbot transcripts to produce standardized verdicts and reports that feed gate decisions.
Run the ocs-chatbot-eval skill against a transcript to generate a verdict YAML and an eval report.
High-intent search queries and answers about installing and using this skill.
Evaluating chatbot transcripts with an LLM involves judging captured QA interactions against a rubric covering correctness, source usage, tone, and tagging to produce machine-readable verdicts and human-readable reports.
Automating QA scoring for chatbot transcripts is done by running an LLM-based verdict engine that reads recent qa-captures and outputs standardized scores alongside per-item judgments to support product decisions.
An LLM verdict engine works by reading captured chatbot transcripts, applying a four-dimension rubric for correctness, source usage, tone, and tagging, and writing structured verdict YAML files to a verdicts directory.
You can generate gate briefs from chatbot QA transcripts by running the evaluation in deep mode, which writes gate-surfaced insights to a gate-briefs directory to support downstream risk and product decisions.
Evaluating chatbot transcripts scores four dimensions: correctness, source usage, tone, and tagging, producing per-item judgments that feed into standardized verdicts and human-readable eval reports.
LLM-based chatbot evaluation requires transcripts stored in the ACE/<opp-name>/qa-captures directory, as the skill reads the most recent captures across quick, deep, and monitor modes to generate verdicts.
dimagi-internalGrade OCS chatbot transcripts against LLM-as-Judge rubrics.
VetCodersAudit agent-delivered work with loctree, semgrep, and cargo clippy checks.
jjacksonCapture structured QA transcripts from ACE OCS chatbot interactions.
aaronjullger-lgtmDesign and validate OSCE clinical simulation flows with grading rubrics and transcript handling.
