Agent Output Quality Verification Engine (품질 검증 엔진)

Automate LLM output quality checks by applying structural and LLM-based validation scripts across pipeline phases like P2 analysis and P3 question generation. The engine assesses dimensions like profile completeness, skill accuracy, and evidence grounding to ensure outputs meet predefined standards.

Automated evidence scoring assigns a 0-100 value to assess LLM output grounding and factual correctness. Based on the score, the system applies revision or rejection gates, triggering specific actions like PASS, REVISE, or REJECT for the generated content.

To validate LLM-generated interview questions, evaluate dimensions including question relevance, clarity, depth, bias, evidence, and factual correctness. The verification engine checks these criteria against predefined standards before allowing pipeline progression.

Yes, you can integrate Langfuse tracing with the automated quality verification pipeline. The engine includes integration points for Langfuse tracing and supports potential database storage of quality metrics generated during the validation process.

Yes, Python scripts are required to implement automated quality gates for LLM outputs. These scripts handle structural and LLM-based validation, executing the automated scoring and revision or rejection logic across the pipeline phases.

You should reject LLM outputs when their evidence score falls into the defined REJECT range, indicating severe grounding or factual correctness failures. Outputs with moderate issues trigger a REVISE action, while high scores pass the quality gate.