agentv-chat-to-eval

To convert chat conversations into evaluation test cases, parse markdown or JSON message arrays to identify test-worthy exchanges, derive evaluation criteria, and generate structured YAML files with `tests:` and `assert:` blocks. This automates regression test creation from successful support logs.

Yes, you can generate multi-turn YAML test cases from a JSON chat log. The conversion process supports multi-turn conversations by identifying key exchanges and producing structured YAML with evaluation criteria, rubrics, and suggested additional evaluators for the AgentV framework.

Yes, the chat to YAML conversion process supports markdown message formats. It accepts chat logs in both Markdown and JSON formats, parsing the unstructured conversations to automatically derive evaluation criteria and generate AgentV-compatible YAML test cases.

The best way to create regression tests from customer support chat logs is to automate the conversion of successful interactions into structured evaluation files. This identifies key problem-solving exchanges and generates YAML with predefined rubrics and assert blocks for your AI agent.

Evaluation criteria derivation works by analyzing chat transcripts to identify test-worthy exchanges, automatically deriving rubrics from the conversation context. It then structures these insights into YAML format with assert blocks and suggests additional evaluators for comprehensive AI agent assessment.