skills.rest
TheodoreGalanosTheodoreGalanosCommunity·6 Agent Skills Included

aec-bench

Create, run, and score engineering benchmark tasks

Creates, runs, and evaluates Architecture, Engineering, and Construction benchmark tasks for AI agents. Generates parameterized task instances from reusable templates, publishes immutable datasets, and scores results with deterministic verifiers. Removes manual experiment setup, fragile scoring scripts, and inconsistent task definitions. Ships guided skills for adding tasks, building templates, configuring experiments, and hardening benchmark quality.
35|5
↗
npx skills add TheodoreGalanos/aec-bench --all -g -y
Available:

Gives the AI agent repository-wide operating rules for AEC-Bench, including benchmark validity priorities, authority order for conflicting documents, safe command usage, and architecture boundaries to respect when changing code.

All Skills in This Repository (6)

Pure Emerald Level Indicators
📦 In Repo
TheodoreGalanosTheodoreGalanos

configure-experiment

Automate interactive AEC experiment configuration and write experiment.yaml.

Community
Advanced
📦 In Repo
TheodoreGalanosTheodoreGalanos

domain-check

Detect cross-domain violations in code changes and generate a structured report.

Community
Advanced
📦 In Repo
TheodoreGalanosTheodoreGalanos

hardening-pass

Detect and fix quality issues in task templates and task instances before benchmarking.

Community
Advanced
📦 In Repo
TheodoreGalanosTheodoreGalanos

create-dataset

Create versioned benchmark datasets through guided template discovery and configuration.

Community
Advanced
📦 In Repo
TheodoreGalanosTheodoreGalanos

add-task

Interview experts to create structured benchmark task seed files.

Community
Advanced
📦 In Repo
TheodoreGalanosTheodoreGalanos

create-template

Convert source_task.json seeds into aec-bench templates with engine.py, params.toml, and instruction.md.

Community
Intermediate

Frequently Asked Questions

FAQPage Schema
How to install aec-bench?▼

Run `npx skills add TheodoreGalanos/aec-bench --all -g -y` in your terminal to install all skills in this suite globally.

What is AEC-Bench used for?▼

It is a Python platform for creating, running, and evaluating Architecture, Engineering, and Construction benchmark tasks that test how well AI agents solve real engineering calculations.

How do I create a new benchmark task?▼

Use the Add Task skill to capture an expert description as a seed file, then the Create Template skill turns it into a reusable generation template with a compute engine, parameters, and instructions.

Which AI models and agents does aec-bench support?▼

It supports tool-loop, PydanticAI, RLM, and direct adapters across Anthropic, Azure OpenAI, Bedrock, and Together providers, with execution through Harbor on Modal, Morph Cloud, or local backends.

Can I compare two models on the same tasks?▼

Yes. Define multiple agents in one experiment.yaml with several repetitions, and AEC-Bench runs both models against identical tasks and produces comparable scored results.

Related Repositories in Software Engineering

View All in Software Engineering→
openclawopenclaw
388.4k

openclaw

Run a personal AI assistant across your devices and chat apps

Official📦 120 Skills
obraobra
280.0k

superpowers

Gives coding agents a disciplined workflow from idea to merged code

Community📦 15 Skills
facebookfacebook
248.4k

react

AI agent skills for building, testing, and porting React core

Official📦 14 Skills