codegen-metrics

Name: codegen-metrics
Availability: InStock
Author: AltairaLabs

Emit structured JSON metrics from codegen evaluation scripts.

9|3|Updated Nov 1, 2025

Search Tags:#automation #metrics #evaluation #json #codegen #scripts

Ready to run

npx skills add https://github.com/AltairaLabs/PromptKit --skill codegen-metrics

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: codegen-metrics
Source: https://github.com/AltairaLabs/PromptKit/tree/main/benchmarks/codegen-eval/skills/codegen-metrics
Command: npx skills add https://github.com/AltairaLabs/PromptKit --skill codegen-metrics

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Soft-metric capture scripts for codegen evals. Emits structured JSON the tool_exec eval handler can capture as EvalResult.Value for jq-based reporting.

Core Features & Use Cases

Host-side measurement scripts for the codegen-eval methodology. These run as session-level evals via tool_exec calling Bash bash /skills/codegen-metrics/scripts/<name>.sh, and the JSON they emit on stdout flows into EvalResult.Value.
Use cases include consistent performance metrics across iterations and easy aggregation in dashboards.

Quick Start

Run the diff_stats.sh script to emit JSON with the line/file deltas of the agent's edits and the total file counts.

Frequently Asked Questions about codegen-metrics

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I capture deterministic codegen evaluation metrics as structured JSON?▼

Deterministic codegen evaluation metrics are captured by running host-side measurement scripts that emit structured JSON on stdout. This output flows directly into EvalResult.Value for jq-based reporting and dashboard aggregation.

How do I measure codegen line and file deltas for agent edits?▼

To measure codegen deltas, run the diff_stats.sh script via a Bash command. It emits JSON containing the line and file deltas of the agent's edits alongside the total file counts.

Can I use bash scripts for automated codegen evaluation in my workflow?▼

Yes, bash scripts automate codegen evaluation by running as session-level evals through tool_exec. The scripts handle metric collection and output machine-readable JSON compatible with reproducible iterations.

What is the best way to aggregate codegen metrics for dashboards?▼

The best way to aggregate codegen metrics for dashboards is emitting structured JSON from evaluation scripts. The JSON output enables consistent performance tracking and easy aggregation across multiple reproducible iterations.

Does codegen evaluation require specific JSON output formats for reporting?▼

Codegen evaluation requires structured JSON output formats for jq-based reporting. The scripts emit machine-readable JSON that the tool_exec eval handler captures as EvalResult.Value for downstream analysis.