openworld-eval

Build and validate open-world evaluation pipelines with reusable benchmark workflows.

Updated Feb 18, 2026

Search Tags:#metrics #calibration #benchmark #evaluate #open-world #abstention

Ready to run

npx skills add https://github.com/rilical/OpenWorld-AI-Image-Detection --skill openworld-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: openworld-eval
Source: https://github.com/rilical/OpenWorld-AI-Image-Detection/tree/main/.agents/skills/openworld-eval
Command: npx skills add https://github.com/rilical/OpenWorld-AI-Image-Detection --skill openworld-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Streamlines evaluation scripting and benchmarking for open-world detectors, enabling consistent metrics and artifact-backed reports across datasets and experiments.

Core Features & Use Cases

Reuse evaluation scripts and metrics functions to export per-sample predictions and selective metrics across datasets.
Provide artifact-backed evaluation flows for multiple benchmarks like CommunityForensics-Small, VCT2, RAID, and ARIA.
Allow reporting selective metrics and empirical coverage with dataset-specific evaluation modes.

Quick Start

Load a saved checkpoint and calibration artifacts from a run directory, then export metrics and per-sample predictions to eval/<dataset>/.

Frequently Asked Questions about openworld-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate open-world detectors across multiple benchmark datasets?▼

Evaluate open-world detectors by loading a saved checkpoint and calibration artifacts from a run directory, then exporting metrics and per-sample predictions to standardized eval paths across datasets.

What is an open-world evaluation pipeline and when do I need abstention metrics?▼

An open-world evaluation pipeline validates detector performance with abstention-application logic, exporting selective metrics and empirical coverage to ensure reliable results on unknown data distributions.

Can I export per-sample predictions for RAID and CommunityForensics benchmarks?▼

Yes, you can export per-sample predictions for RAID, CommunityForensics-Small, VCT2, and ARIA datasets by reusing the provided evaluation scripts and metric functions.

How do I maintain consistent evaluation contracts for open-world detection experiments?▼

Maintain consistent evaluation contracts by applying a uniform benchmark workflow that coordinates calibration artifacts and dataset metadata, enabling artifact-backed reports for reliable experiment results.

Does this benchmark workflow support selective metric reporting and empirical coverage?▼

Yes, the benchmark workflow supports reporting selective metrics and empirical coverage through dataset-specific evaluation modes, generating artifact-backed reports for open-world detection runs.