majesticway-task-creation

You can generate AEC benchmark tasks by creating standardized instances from MajesticWay project data, covering scopes like intrasheet and intraproject, to evaluate information-retrieval and multimodal AI agents on construction documents.

Synthetic defect injection involves defining errors like mismatched dimensions within source PDFs and generating the required task.toml, instruction.md, and test.sh files to test if AI agents can detect the injected construction document defects.

Formatting construction document QA tasks for Harbor benchmarks requires setting up the correct file structure and authoring verifier scripts, followed by local task validation to ensure consistency and evaluability for AI agent testing.

Yes, you can design task instances for cross-reference resolution and spec-drawing synchronization, allowing you to test whether an AI agent can detect mismatches between employer requirements and construction drawings.

Local validation against benchmark conventions requires generating specific files including task.toml, instruction.md, and test.sh, while ensuring the overall file structure properly supports the defined task scopes and injected construction defects.

A standardized format eliminates the time-consuming manual formatting process, ensuring consistency and evaluability across different test instances when evaluating multimodal AI agents on real construction project documents.