skills.rest
rilical
rilicalCOMMUNITY
Explore all skills by this author →
GitHub

openworld-eval

Build and validate open-world evaluation pipelines with reusable benchmark workflows.

Updated Feb 18, 2026
Search Tags:#metrics#calibration#benchmark#evaluate#open-world#abstention
One-click install
npx skills add https://github.com/rilical/OpenWorld-AI-Image-Detection --skill openworld-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openworld-eval
Source: https://github.com/rilical/OpenWorld-AI-Image-Detection/tree/main/.agents/skills/openworld-eval
Command: npx skills add https://github.com/rilical/OpenWorld-AI-Image-Detection --skill openworld-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Streamlines evaluation scripting and benchmarking for open-world detectors, enabling consistent metrics and artifact-backed reports across datasets and experiments.

Core Features & Use Cases

  • Reuse evaluation scripts and metrics functions to export per-sample predictions and selective metrics across datasets.
  • Provide artifact-backed evaluation flows for multiple benchmarks like CommunityForensics-Small, VCT2, RAID, and ARIA.
  • Allow reporting selective metrics and empirical coverage with dataset-specific evaluation modes.

Quick Start

Load a saved checkpoint and calibration artifacts from a run directory, then export metrics and per-sample predictions to eval/<dataset>/.

Frequently Asked Questions about openworld-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate open-world detectors across multiple benchmark datasets?▼

Evaluate open-world detectors by loading a saved checkpoint and calibration artifacts from a run directory, then exporting metrics and per-sample predictions to standardized eval paths across datasets.

What is an open-world evaluation pipeline and when do I need abstention metrics?▼

An open-world evaluation pipeline validates detector performance with abstention-application logic, exporting selective metrics and empirical coverage to ensure reliable results on unknown data distributions.

Can I export per-sample predictions for RAID and CommunityForensics benchmarks?▼

Yes, you can export per-sample predictions for RAID, CommunityForensics-Small, VCT2, and ARIA datasets by reusing the provided evaluation scripts and metric functions.

How do I maintain consistent evaluation contracts for open-world detection experiments?▼

Maintain consistent evaluation contracts by applying a uniform benchmark workflow that coordinates calibration artifacts and dataset metadata, enabling artifact-backed reports for reliable experiment results.

Does this benchmark workflow support selective metric reporting and empirical coverage?▼

Yes, the benchmark workflow supports reporting selective metrics and empirical coverage through dataset-specific evaluation modes, generating artifact-backed reports for open-world detection runs.

Related Skills

Explore More Data & Analytics Skills→
rilicalrilical

calibration-conformal

Calibrate temperature scaling and conformal prediction artifacts for detector abstention.

Community
Advanced
tbroadleytbroadley

read-inspect-eval

Extract samples, messages, events, and metrics from Inspect AI .eval logs.

Community
Intermediate
sinfinysinfiny
5

autonomous-webserial-loop

Automate iterative planning, drafting, and revision of mystery webserial chapters.

Community
Advanced
surus-latsurus-lat
8

push-to-latamboard

Merge local benchmark results into the HuggingFace LatamBoard leaderboard dataset.

Official
Advanced