ultraqa

Runs adversarial end-to-end QA cycles with hostile scenario matrices, temporary harnesses, and structured evidence reports.

Updated Feb 1, 2024
One-click install
npx skills add https://github.com/ogiboy/portfolio-app --skill ultraqa-ogiboy
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ultraqa
Source: https://github.com/ogiboy/portfolio-app/tree/main/.codex/skills/ultraqa
Command: npx skills add https://github.com/ogiboy/portfolio-app --skill ultraqa-ogiboy

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Standard build, lint, and test checks can pass while real behavior is still broken. UltraQA closes that gap by forcing adversarial dynamic end-to-end verification—malformed input, prompt injection, interruptions, stale state, flaky tests, and misleading success output—before declaring a goal met. ## Core Features & Use Cases - Adversarial Scenario Matrix: Tracks normal, hostile, malformed, interruption, injection, cancel/resume, stale-state, dirty-worktree, hung-command, flaky, and misleading-output scenarios with evidence and cleanup status. - Bounded Test-Diagnose-Fix Cycles: Repeats up to 5 cycles of plan, baseline verification, adversarial e2e execution, architect diagnosis, targeted fixes, and cleanup, with early exit on repeated failures or safety boundaries. - Temporary Harness Guardrails: Generates throwaway test harnesses with absolute imports, safe file writers, sanitized environment variables, and mandatory cleanup so harness debris never pollutes the worktree. - Use Case: After changing a CLI's resume logic, run UltraQA to verify that stale state files, repeated continue commands, and a CLI printing SUCCESS while exiting 1 are all caught, fixed, and reported with evidence. ## Quick Start Ask the agent to run UltraQA against your test suite with a goal such as verifying that the changed behavior survives hostile inputs, interruptions, and misleading success output.

Frequently Asked Questions about ultraqa

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run adversarial end-to-end QA on my code changes?

Invoke UltraQA with a goal flag such as --tests, --build, or --custom. It builds a scenario matrix covering hostile and edge-case behaviors, runs baseline verification plus adversarial e2e scenarios, then diagnoses, fixes, and retests until the goal is met or a stop condition is reached.

What is the difference between UltraQA and running npm test?

Running npm test only checks existing tests, while UltraQA treats baseline commands as necessary but insufficient evidence. It additionally generates hostile scenarios like prompt injection, stale state, and misleading success output, and refuses to declare success without that adversarial coverage.

Can UltraQA test CLI tools and interactive services?

Yes, the --interactive goal type targets CLI and service behavior using a qa-tester style harness. It exercises normal, hostile, malformed, interruption, resume, and hung-command interactions with bounded timeouts and exit-code validation.

When does UltraQA stop before the goal is met?

It stops after 5 cycles, when the same failure occurs 3 times, when a safety boundary blocks a scenario, or on environment errors like hung commands. Each stop produces a structured report with failures, fixes attempted, residual risks, and cleanup status.

Does UltraQA clean up temporary test harnesses and state?

Yes, every generated harness, fixture, log, and spawned process is recorded in the scenario matrix and removed during cleanup unless intentionally kept. It also clears its tracked state via the omx state CLI and verifies worktree status before and after.