create-inspect-task

Create custom inspect-ai evaluation tasks with experiment-guided and standalone modes.

16|1|Updated Apr 4, 2025

Search Tags:#evaluation #fine-tuning #workflow-automation #ai-testing #task-creation #inspect-ai

Ready to run

npx skills add https://github.com/niznik-dev/cruijff_kit --skill create-inspect-task

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: create-inspect-task
Source: https://github.com/niznik-dev/cruijff_kit/tree/main/.claude/skills/create-inspect-task
Command: npx skills add https://github.com/niznik-dev/cruijff_kit --skill create-inspect-task

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires yaml.

What problem does it solve?

This Skill eliminates the complexity and time-consuming process of manually creating custom AI evaluation tasks, allowing you to focus on research insights rather than boilerplate code.

Core Features & Use Cases

Experiment-Guided Creation: Automatically extracts configuration from existing experiments to pre-populate evaluation settings.
Standalone Task Builder: Guides users through creating evaluation tasks from scratch with best practices.
Use Case: When you've completed a fine-tuning experiment and need to evaluate model performance, this Skill creates production-ready evaluation scripts in minutes instead of hours.

Quick Start

Use the create-inspect-task skill to build a custom evaluation task for my sentiment classification dataset.

Frequently Asked Questions about create-inspect-task

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate AI evaluation task creation for fine-tuning experiments?▼

Create custom inspect-ai evaluation tasks automatically by extracting configuration from existing experiments. The Skill guides you through dataset, model, and objective setup, then generates production-ready task files and documentation in minutes instead of hours.

Can I build evaluation tasks from scratch without an existing experiment?▼

Yes. The Standalone Task Builder mode walks you through creating evaluation tasks from the ground up with best practices, including solver chain design, scoring selection, and parameterization, without requiring prior experiment data.

What does the Skill generate as output?▼

It produces a complete runnable task file, accompanying documentation, a create-inspect-task.log for tracking, and clear usage instructions. All code is ready to execute in your evaluation pipeline immediately.

How does this Skill handle experiment configuration extraction?▼

The Experiment-Guided mode automatically parses experiment_summary.yaml files to pre-populate evaluation settings, eliminating manual configuration entry and reducing setup time.

What's the best way to transition from evaluation task creation to model scoring?▼

After generating your evaluation task with scoring configuration, you can immediately run inspect-ai evaluations against your models to collect performance metrics and iterate on task parameters.