nodefony-devkit-bench

Benchmarks Nodefony scaffold output through compilation, HTTP, and agent discoverability tests.

Updated Dec 19, 2023
One-click install
npx skills add https://github.com/nodefony/nodefony-core --skill nodefony-devkit-bench-nodefony
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: nodefony-devkit-bench
Source: https://github.com/nodefony/nodefony-core/tree/main/.claude/skills/nodefony-devkit-bench
Command: npx skills add https://github.com/nodefony/nodefony-core --skill nodefony-devkit-bench-nodefony

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? String-assertion tests on generated templates cannot prove that scaffolded Nodefony code actually compiles, runs, or is discoverable by AI agents. This Skill provides three complementary benches that measure what the scaffold truly produces: whether generated code stands up (compilation, tests, real HTTP), whether an agent dropped into a fresh app finds the tooling instead of guessing, and whether a real-world data model is expressible with the field grammar. ## Core Features & Use Cases - Truth bench (verify-generated.mjs): Builds an isolated witness application from tarballs and exercises generation, compilation, lint, real HTTP (201/422/409/pagination), production mode, and multi-database dialects (SQLite, PostgreSQL, MySQL/MariaDB). - Discoverability bench (bench-discoverability.mjs): Runs real agents (Claude, Vibe, Codex, Gemini) through frozen tasks in a reset decor, judging them with objective probes and gates rather than LLM judges, and measures turns, duration, and cost. - Transcript analysis (analyse-transcript.mjs): Parses transcripts from six agent dialects to report turns, cost, MCP calls, failed gestures, and full timelines. - Use Case: After modifying a Nodefony generator template, run the truth bench to confirm the generated entity compiles and its REST routes respond correctly before declaring the change done. ## Quick Start Run the devkit bench to verify that the Nodefony scaffold still generates code that compiles and passes real HTTP checks.

Frequently Asked Questions about nodefony-devkit-bench

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify that Nodefony generated code actually compiles and runs?

Run node .claude/skills/nodefony-devkit-bench/scripts/verify-generated.mjs after building the checkout. It creates an isolated witness app from tarballs, generates entities and modules, then checks compilation, lint, tests, and real HTTP responses.

How do I test whether an AI agent discovers Nodefony generators?

Run bench-discoverability.mjs, which executes frozen tasks with real agents in a reset decor. Objective probes and gates judge the transcript and git diff, and analyse-transcript.mjs reports turns, duration, cost, and MCP calls.

Does the bench support PostgreSQL and MySQL in addition to SQLite?

Yes, pass --database postgres, mysql, or mariadb to verify-generated.mjs. Each engine recreates the witness application because the dialect is fixed at creation time, and server engines require three pre-created databases.

Why do string-assertion tests miss scaffold defects?

Assertions only check that rendered files contain expected strings. Real defects found by the bench include generated types that fail compilation, ORM boot errors from unregistered relation targets, and foreign key type mismatches rejected by PostgreSQL.

Which AI agents can the discoverability bench run?

The bench supports Claude (default), Vibe, Codex, and Gemini, configured via NF_DEVKIT_BENCH_AGENT and NF_DEVKIT_BENCH_AGENT_ARGS. Effort metrics like turns and cost are only emitted by the Claude CLI format; other agents are judged by probes.

Why does a bench run show zero MCP calls for an agent?

Zero MCP calls usually mean the porte was never declared, not that the agent chose the CLI. Check the declaration constat printed by the bench, since silent declaration failures leave the agent without tools.