ultraqa

Runs adversarial end-to-end QA cycles with hostile scenario matrices and structured failure reports.

10|1|Updated Jul 7, 2026
One-click install
npx skills add https://github.com/catalystctl/catcode --skill ultraqa-catalystctl
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ultraqa
Source: https://github.com/catalystctl/catcode/tree/main/.omp/skills/ultraqa
Command: npx skills add https://github.com/catalystctl/catcode --skill ultraqa-catalystctl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Green unit tests and shallow build/lint/typecheck checklists often miss real behavior failures such as prompt injection, stale state, misleading success output, and hung commands. This Skill closes that gap by generating adversarial end-to-end scenarios, diagnosing root causes, and cycling test-fix-retest until the goal is verifiably met. ## Core Features & Use Cases - Adversarial Scenario Matrix: Maintains a structured matrix covering malformed input, prompt injection, cancel/resume, stale state, dirty worktrees, hung commands, flaky tests, and misleading success output. - Test-Diagnose-Fix-Retest Loop: Runs up to 5 bounded cycles with a fresh reviewer subagent for root-cause diagnosis, stopping on success, repeated failure signatures, or safety bounds. - Temporary Harness Generation: Creates throwaway test harnesses and fixtures under a dedicated directory, records every artifact, and cleans them up after verification. - Structured Reporting: Produces a report with commands run, failures found, fixes applied, cleanup status, and residual risks. - Use Case: After implementing a CLI resume feature, invoke UltraQA to verify it handles interrupted runs, stale session files, and injection attempts before shipping. ## Quick Start Ask the agent to run ultraqa against the current changes with a goal such as verifying that the build passes and the new resume behavior survives adversarial end-to-end scenarios.

Frequently Asked Questions about ultraqa

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run adversarial end-to-end tests on my code changes?

Invoke the skill with a goal flag such as --tests, --build, or --custom. It builds a scenario matrix covering malformed input, injection, and stale state, then runs baseline commands plus adversarial harnesses and reports failures with evidence.

What is the difference between adversarial e2e testing and unit tests?

Unit tests verify isolated logic under expected conditions, while adversarial e2e testing models hostile users, interruptions, and corrupted state against the running system. This skill treats green unit tests as insufficient and requires behavior-level scenario evidence.

How do I test prompt injection resistance in an AI agent?

The scenario matrix includes prompt injection rows covering override instructions, secret exfiltration attempts, and false success claims. Each row runs a harness with an expected signal, and results are recorded with evidence in the report.

Does this QA workflow handle flaky tests and hung commands?

Yes. Flaky test rows use rerun strategies and failure clustering to avoid false greens from one lucky pass. Hung commands are bounded by timeouts with killed-child cleanup and recovery notes.

When does the QA loop stop and what are its limits?

The loop stops when the goal is met, after 5 cycles, when the same failure signature appears 3 times, or when a safety bound blocks progress. Each stop condition produces a structured exit message with residual risks.