skills.rest
numerai
numeraiOFFICIAL
Explore all skills by this author →
WebsiteGitHub

numerai-experiment-design

Plan and execute round-based Numerai experiments with documented decision tracking.

1.2k|310|Updated Jan 6, 2017
Search Tags:#benchmark#reproducibility#model-evaluation#machine-learning#experiment-design#Numerai
One-click install
npx skills add https://github.com/numerai/example-scripts --skill numerai-experiment-design
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: numerai-experiment-design
Source: https://github.com/numerai/example-scripts/tree/main/numerai/agents/skills/numerai-experiment-design
Command: npx skills add https://github.com/numerai/example-scripts --skill numerai-experiment-design

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a structured workflow to design, run, and document Numerai experiments, enabling disciplined, iterative research across rounds.

Core Features & Use Cases

  • Structured experiment planning: define baseline, feature_set, metrics, and decision rules for progression.
  • Round-based evaluation: run multiple configs per round, synthesize results, and decide next steps.
  • Reproducible experiments: maintain an experiment.md with decisions, results, and rationale to ensure traceability.

Quick Start

Create a new experiment directory under agents/experiments (one line per idea). Initialize baseline and initial configs, then run the training and evaluation loop with PYTHONPATH=numerai python3 -m agents.code.modeling --config <config> --output-dir <experiment_dir>. Update experiment.md after each round with results and decisions.

Frequently Asked Questions about numerai-experiment-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design reproducible machine learning experiments for Numerai models?▼

Running Numerai experiments requires setting PYTHONPATH=numerai and executing the modeling module with specific config and output directory parameters to train and evaluate model configurations per round.

What is the best way to manage round-based model evaluation configs?▼

To document experiment rationale for machine learning reproducibility, maintain an experiment.md file in your experiment directory and update it after each round with results and progression decisions.

Do I need to install specific dependencies to run Numerai experiment workflows?▼

You need a Python environment configured to run Numerai workflows, specifically requiring PYTHONPATH=numerai to execute the provided training and evaluation pipeline for your model ideas.

How does structured experiment planning improve model benchmarking?▼

Structured experiment planning improves model benchmarking by enforcing a repeatable workflow that defines baselines and decision rules, ensuring disciplined evaluation of feature sets across multiple configurations.

Related Skills

Explore More Data & Analytics Skills→
donzales12donzales12

numerai-model-implementation

Register and integrate a new Numerai model type into the training pipeline.

Community
Intermediate
donzales12donzales12

numerai-experiment-design

Plan and organize Numerai experiment rounds with baselines, feature sets, and evaluation criteria.

Community
Advanced
numerainumerai
1.2k

numerai-model-upload

Generate a portable Numerai model-upload pickle with a self-contained predict function.

Official
Advanced
informatico-madridinformatico-madrid
1

mlflow

Manages ML lifecycle with experiment tracking, model registry, and deployment tools for TensorFlow, PyTorch, Scikit-Learn.

Community
Advanced