codec-conformance-eval

What problem does it solve?

Ensures C reference model decoders correctly implement codec standards by automating the tedious process of building the ref C decoder, running hundreds of official conformance bitstreams, and verifying bitexact or quality-based matches against golden outputs so algorithmic bugs are detected early.

Core Features & Use Cases

• Automated Build: Builds ref C decoder sources (C11, gcc) with Makefile fallback and configurable C flags.
• Parallel Execution: Runs large conformance sets locally with ProcessPoolExecutor or submits jobs to AWS Batch for large-scale evaluation.
• Flexible Comparison Modes: Supports MD5, byte-for-byte bitexact, PSNR threshold checks, and opt-in SSIM/VMAF quality metrics.
• Reporting & Coverage: Produces per-stream PASS/FAIL reports, profile/level coverage matrix, and a templated conformance report for documentation.
• Use Cases: Algorithm verification during Research/Architecture phases, pre-RTL conformance gatekeeping, and third-party conformance validation.

Quick Start

Run the codec-conformance-eval skill to build the reference C decoder, run parallel MD5 or bitexact conformance checks for the chosen standard/profile, and generate the conformance report.