genai-evaluation-metrics

What problem does it solve?

Provides a coherent, production-minded workflow to evaluate generative models by selecting and computing the right mixture of distributional, perceptual, diversity, and memorization metrics so teams can trust reported quality and detect mode collapse or data copying.

Core Features & Use Cases

• Comprehensive metric catalog: Guidance and implementations for FID, sFID, FDD (DINOv2), FVD (I3D), IS, KID, PRDC, LPIPS, SSIM, AuthPct, Vendi, and FD-infinity for both images and videos.
• Operational patterns: Metric orchestrator pattern, DINOv2 multi-metric extraction, online (training-time) vs offline benchmarking strategies, sample-count guidance, and multi-GPU gathering with memory management best practices.
• Real-world example: Run 5k-sample online evaluation during training using the EMA model, DINOv2 features for texture-sensitive FDD and a combined PRDC/IS pass to detect mode collapse and track best checkpoints.

Quick Start

Evaluate your model's FID, KID, LPIPS, and AuthPct on a held-out validation set using DINOv2 features and 5000 samples.