karpathy-metric-pre

What problem does it solve?

It helps you stress-test an optimization metric to uncover how it can be gamed, contaminated, or diverge from the real business outcome it is supposed to represent.

Core Features & Use Cases

• Adversarial metric pre-mortem: Enumerates concrete ways an optimizer can inflate the metric while delivering less real value (e.g., gaming evaluation hooks or measurement edge cases).
• Defense design with countermetrics: Proposes secondary metrics, holdout scenarios, and a disappearance test to detect brittle improvements.
• Evaluation diversity plan: Produces a single actionable document to run an adversarial and periodic test regimen across optimization cycles.

Quick Start

Ask the AI to run a metric gaming pre-mortem by providing the primary metric definition, intended business outcome, what the optimization agent can edit, and how the evaluation is currently performed.