configure-experiment

Configuring a benchmark experiment involves interactively discovering available tasks, selecting disciplines and difficulties, setting up agents and compute, and outputting a ready-to-run experiment.yaml validated against the ExperimentManifest schema.

Planning an AI agent benchmark run requires reading project context like aec-bench.toml and task.toml files to identify available datasets, then configuring agents and generating a structured experiment manifest for execution.

Yes, you can configure multiple AI agents for model comparison within a single experiment setup. The configuration process supports adding various agents and provides a dry-run preview of the trial plan before finalizing the manifest.

You discover available tasks and datasets by scanning task.toml files across the project directory. This process reports all accessible disciplines, difficulty levels, and dataset availability for the experiment configuration.

Yes, the generated YAML output is strictly validated against the ExperimentManifest schema. This ensures the final experiment.yaml written to the project root has an exact structure and is ready for benchmark execution.

Setting up a benchmark experiment requires project context files like aec-bench.toml for global settings and task.toml files for defining individual benchmark tasks, along with verifiable dataset availability within the project.