skills.rest
AltairaLabs
AltairaLabsOFFICIAL
Explore all skills by this author →
GitHub

codegen-metrics

Emit structured JSON metrics from codegen evaluation scripts.

9|3|Updated Nov 1, 2025
Search Tags:#automation#metrics#evaluation#json#codegen#scripts
One-click install
npx skills add https://github.com/AltairaLabs/PromptKit --skill codegen-metrics
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: codegen-metrics
Source: https://github.com/AltairaLabs/PromptKit/tree/main/benchmarks/codegen-eval/skills/codegen-metrics
Command: npx skills add https://github.com/AltairaLabs/PromptKit --skill codegen-metrics

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Soft-metric capture scripts for codegen evals. Emits structured JSON the tool_exec eval handler can capture as EvalResult.Value for jq-based reporting.

Core Features & Use Cases

  • Host-side measurement scripts for the codegen-eval methodology. These run as session-level evals via tool_exec calling Bash bash /skills/codegen-metrics/scripts/<name>.sh, and the JSON they emit on stdout flows into EvalResult.Value.
  • Use cases include consistent performance metrics across iterations and easy aggregation in dashboards.

Quick Start

Run the diff_stats.sh script to emit JSON with the line/file deltas of the agent's edits and the total file counts.

Frequently Asked Questions about codegen-metrics

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I capture deterministic codegen evaluation metrics as structured JSON?▼

Deterministic codegen evaluation metrics are captured by running host-side measurement scripts that emit structured JSON on stdout. This output flows directly into EvalResult.Value for jq-based reporting and dashboard aggregation.

How do I measure codegen line and file deltas for agent edits?▼

To measure codegen deltas, run the diff_stats.sh script via a Bash command. It emits JSON containing the line and file deltas of the agent's edits alongside the total file counts.

Can I use bash scripts for automated codegen evaluation in my workflow?▼

Yes, bash scripts automate codegen evaluation by running as session-level evals through tool_exec. The scripts handle metric collection and output machine-readable JSON compatible with reproducible iterations.

What is the best way to aggregate codegen metrics for dashboards?▼

The best way to aggregate codegen metrics for dashboards is emitting structured JSON from evaluation scripts. The JSON output enables consistent performance tracking and easy aggregation across multiple reproducible iterations.

Does codegen evaluation require specific JSON output formats for reporting?▼

Codegen evaluation requires structured JSON output formats for jq-based reporting. The scripts emit machine-readable JSON that the tool_exec eval handler captures as EvalResult.Value for downstream analysis.

Related Skills

Explore More Software Engineering Skills→
bhutanobhutano
24

usage-bar

Install and configure a persistent Claude Code usage bar across platforms.

Community
Intermediate
tstaplertstapler
8

github-pr

Automate GitHub pull request operations with gh CLI and jq.

Community
Intermediate
lvndrylvndry
47

obsidian

Automate Obsidian note creation, retrieval, and vault organization via obsidian-cli.

Community
Advanced
jerrywu001jerrywu001
6

benchmark

Benchmark web pages by comparing navigation metrics and Core Web Vitals.

Community
Advanced