eng-real-scenario-qa

Runs release-grade QA by executing autonomous startup scenarios on the Compozy runtime.

2.7k|171|Updated Mar 28, 2026
One-click install
npx skills add https://github.com/compozy/compozy --skill eng-real-scenario-qa
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: eng-real-scenario-qa
Source: https://github.com/compozy/compozy/tree/main/.agents/skills/eng/eng-real-scenario-qa
Command: npx skills add https://github.com/compozy/compozy --skill eng-real-scenario-qa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Validating an agent runtime with mocks and unit tests misses real integration failures. This Skill dogfoods Compozy by running a full fictional startup project with live providers, observing agent behavior across CLI, API, Web, and runtime surfaces, and auditing the evidence against a strict contract.

Core Features & Use Cases

  • Playbook-driven scenarios: Selects and rotates startup playbooks (e.g., northstar-pay, devtool-oss-launch) that define agents, channels, task trees, required deliverables, and collaboration minimums.
  • Single-kickoff autonomy: Posts exactly one in-persona operator kickoff behind a scheduler barrier, then observes without ever prompting agents again, so stalls surface as real bugs.
  • Strict evidence audit: The auditor enforces compiled/parsed deliverables, peer messages, review cycles, disagreement resolution, and forbidden-prompt-phrase scanning, exiting with blocking codes on failure.
  • Use Case: Before a Compozy release, run the skill with a playbook slug to bootstrap an isolated lab, let agents build real product artifacts for 30 minutes, then audit the evidence and file runtime bugs for any stalls or missing deliverables.

Quick Start

Run the eng-real-scenario-qa skill with playbook northstar-pay to execute a full release-grade QA scenario against the Compozy runtime.

Frequently Asked Questions about eng-real-scenario-qa

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a real-scenario QA test on the Compozy runtime?

Invoke the skill with an optional playbook slug such as northstar-pay. It bootstraps an isolated lab via eng-qa-bootstrap, posts one operator kickoff, observes the runtime for a configured window, and audits the evidence with a strict auditor script.

What is a playbook in Compozy real-scenario QA?

A playbook is a markdown spec under references/playbooks defining the startup scenario: agents, channels, task tree, required deliverables, collaboration minimums, and the kickoff brief. When no slug is given, the skill rotates from the previous run's recorded playbook ref.

Why does the QA auditor reject prompts containing phrases like 'you are the QA'?

Forbidden phrases tell agents they are being evaluated, which changes their behavior toward producing evaluator-shaped evidence instead of real product work. The auditor scans all prompts and exits with a blocking code 2 on any match, requiring the playbook's kickoff_brief to be rewritten.

What happens when the runtime observer detects a stall?

The observer exits with code 1 and records the unchanged active tasks and loop runs in observation-summary.json. The skill then files a mandatory runtime bug naming the silent agent and stalled task, without ever sending another prompt to the agents under test.

When should I not use real-scenario QA?

Do not use it for smoke tests, static checks, mock-only validation, or unit-test work, as stated in the skill description. It is designed only for release-grade or complex-integration QA with live providers and full cross-surface evidence.

Does the QA lab require cleanup after a run?

Yes, teardown is mandatory on every terminal verdict including PASS, FAIL, BLOCKED, or abort. The skill runs the bootstrap manifest's exact teardown command and requires teardown.json to report clean true, with surviving processes treated as a blocking failure.