skills.rest
maximhq
maximhqOFFICIAL
Explore all skills by this author →
WebsiteGitHub

harness-test-writer

Adds regression test cases to the Bifrost Postman provider harness from merged PRs or GitHub issues.

7.7k|1.1k|Updated Mar 19, 2025
Search Tags:#regression testing#Postman#gitHub#newman#API gateway#Code review#Test automation
One-click install
npx skills add https://github.com/maximhq/bifrost --skill harness-test-writer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: harness-test-writer
Source: https://github.com/maximhq/bifrost/tree/main/.claude/skills/harness-test-writer
Command: npx skills add https://github.com/maximhq/bifrost --skill harness-test-writer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a bug fix or feature lands in Bifrost, the corresponding wire-level behavior often lacks regression coverage in the provider harness, so regressions can silently return. This Skill turns a merged PR or GitHub issue into properly formatted test cases in the Postman collection at tests/e2e/api/collections/provider-harness.json.

Core Features & Use Cases

  • Reference Resolution: Fetches PRs or issues via the GitHub CLI, including linked issues that contain the client-visible reproduction details.
  • Wire-Path Tracing: Reads the changed code to identify affected routes, providers, request shapes, and the observable failure signature to pin.
  • Coverage Gap Analysis: Searches the existing collection and HARNESS_COVERAGE_BACKLOG.md to avoid duplicating existing tests.
  • Surgical Insertion: Appends a new top-level folder textually without reformatting the 1.5MB JSON file, then validates with the augment and filter scripts.
  • Use Case: After PR #1234 fixes a role-stripping bug in chat-to-responses conversion, run the Skill with the PR number to generate streaming and non-streaming regression cases that pin the fixed behavior.

Quick Start

Invoke the Skill with a PR number, issue number, or GitHub URL, for example by asking to add harness regression coverage for PR 1234 in the bifrost repository.

Frequently Asked Questions about harness-test-writer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add regression tests to the Bifrost provider harness?▼

Invoke the Skill with a PR number, issue number, or GitHub URL. It fetches the reference, traces the affected wire path, checks existing coverage, designs cases following harness conventions, and inserts them into tests/e2e/api/collections/provider-harness.json.

What is the difference between the provider harness and the Go test harness in Bifrost?▼

The provider harness is a Postman collection run via make run-provider-harness-test that tests Bifrost externally over HTTP. The Go harness in core/internal/llmtests runs via make test-core and tests Go-level scenarios directly.

Does the harness test writer run the live test suite automatically?▼

No. The live suite starts a gateway and makes paid provider API calls, so the Skill only validates the collection with the augment and filter scripts. It reports the make run-provider-harness-test command and offers to run it.

Why must provider-harness.json not be rewritten with JSON.stringify?▼

The roughly 1.5MB collection file is not byte-stable under JSON parse and re-serialize due to escape differences. The Skill appends new folders textually before the closing bracket so the diff contains only added lines.

What happens if harness coverage already exists for a PR or issue?▼

The Skill searches the collection for the feature's keywords before writing anything. If matching coverage already exists, it reports where the coverage is and stops without adding duplicate cases.

Related Skills

Explore More Software Engineering Skills→
gobifrostgobifrost
55

bifrost-issues

Automate GitHub Issue tracking, git worktrees, and PR merges for Bifrost.

Official
Advanced
0xHoneyJar0xHoneyJar
4

eval-running

Execute Loa framework evaluation suites and output JSON results.

Official
Intermediate
leasebaseleasebase

bff-gateway

Orchestrate frontend API requests and aggregate microservice responses in the BFF gateway.

Official
Intermediate
thrawn01thrawn01
11

surface-testing

Test HTTP endpoints, CLIs, and library exports without internal calls.

Community
Advanced