evaluator

Evaluate rendered UGC videos and generate eval.json and eval-report.md.

118|12|Updated May 5, 2026
One-click install
npx skills add https://github.com/alecs5am/ralphy --skill evaluator-alecs5am
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: evaluator
Source: https://github.com/alecs5am/ralphy/tree/main/.agents/skills/evaluator
Command: npx skills add https://github.com/alecs5am/ralphy --skill evaluator-alecs5am

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Quickly assess the quality and integrity of rendered UGC videos by producing a structured evaluation report that highlights scene segmentation, audio, captions, and visual consistency. The fixer agent can read the report to apply targeted fixes.

Core Features & Use Cases

  • Automated quality evaluation across render modes (structure, keyframe, native-video, deep-style) and generation of eval.json + eval-report.md for downstream agents.
  • Outputs deterministic findings and tailored handoffs for fixers, with references to scenario.json, BRIEF.md, STYLE_LOCK.md when present.
  • Supports triggering phrases like "evaluate this video" or "is this ready to ship" to kick off the evaluation workflow.

Quick Start

Run the evaluator on a rendered mp4 using ralphy eval video <path-to-mp4>, then review the generated eval.json and eval-report.md for the results.

Frequently Asked Questions about evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate video quality evaluation for rendered mp4 files?

Automate video quality evaluation by running the evaluator on a rendered mp4 to generate an eval.json and eval-report.md highlighting scene segmentation, audio, captions, and visual consistency.

Can I run mid-render quality checks on UGC videos before final cuts?

Yes, you can run mid-render quality checks on UGC videos. The evaluator assesses quality across structure, keyframe, native-video, and deep-style render modes for both final cuts and intermediate renders.

Does the video evaluation report support handoffs to downstream fixer agents?

Yes, the video evaluation report supports downstream fixer agents by outputting deterministic findings and tailored handoffs in a machine-readable eval.json contract for targeted automated fixes.

What project context files can I use to improve UGC scene analysis accuracy?

You can improve UGC scene analysis accuracy by optionally incorporating project context from scenario.json, BRIEF.md, and STYLE_LOCK.md files when evaluating rendered video outputs.

What is the best way to generate a machine-readable eval.json contract for artifact detection?

The best way to generate a machine-readable eval.json contract for artifact detection is using an automated evaluation tool that analyzes rendered video outputs and surfaces actionable findings.