ek-dev-add-metric

To add custom reference-based evaluation metrics, you wrap external scoring libraries into a standardized Protocol and register them within the ek ecosystem. This ensures mathematical correctness and consistency for your offline benchmarking tasks.

A metric Protocol is a strict typing interface requiring a specific signature of (pred, gold, *, grammar). It enforces standardization across reference-based evaluation metrics, ensuring all custom scoring functions conform to the ek framework's required structure.

You wrap external libraries by implementing them into the standardized metric Protocol and registering them for both type-based dispatch and explicit name-based resolution. This enables domain-specific scoring for strings, fields, tables, and typed graphs.

Yes, it supports developing specialized metrics like cost-weighted typed-graph edit distance and custom table-structure metrics. It guides you through the required contract, dependency gating, and canonicalization standards for these offline evaluations.

Yes, implementing offline evaluation metrics requires strict adherence to canonicalization standards and cost-weighting conventions. These standards ensure your custom metrics integrate correctly with the ek score and evaluate facades.

Registering custom metrics requires dependency gating and adherence to the ek registry architecture. Limitations include ensuring your external libraries conform to the strict typing Protocol and maintain mathematical correctness for offline evaluation.