skills.rest
werf
werfOFFICIAL
Explore all skills by this author →
Website𝕏@werf_ioGitHub

test-the-tests

Validates tests by applying real code mutations and confirming they fail as expected.

4.7k|238|Updated Jan 25, 2016
Search Tags:#test quality#regression testing#mutation testing#test validation#falsifiability#Code review
One-click install
npx skills add https://github.com/werf/werf --skill test-the-tests
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-the-tests
Source: https://github.com/werf/werf/tree/main/.agents/skills/test-the-tests
Command: npx skills add https://github.com/werf/werf --skill test-the-tests

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Passing tests and high coverage do not prove a test actually catches bugs. This Skill verifies that a test genuinely discriminates correct from incorrect behavior by deliberately mutating the implementation and confirming the test fails for the right reason.

Core Features & Use Cases

  • Mutation Loop: Applies the smallest plausible fault (inverted condition, removed validation, suppressed error) to the implementation, runs the test, and confirms it fails for the right reason before reverting.
  • Weak-Test Detection: Identifies tests that assert implementation trivia, re-record the same event, or hold true even under the bug they claim to guard.
  • Safe Revert Discipline: Enforces copy-based backup and restore so uncommitted work is never destroyed by git checkout during mutation.
  • Use Case: After writing a regression test for a bug fix, run the mutation loop to reintroduce the original bug and confirm the new test actually fails, proving it guards the fix.

Quick Start

Ask the AI to verify that the test you just wrote actually fails when the bug it guards against is reintroduced into the implementation.

Frequently Asked Questions about test-the-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify a test actually catches the bug it covers?▼

Apply the smallest plausible mutation to the implementation, such as reverting to the prior buggy behavior, then run the test and confirm it fails for the right reason. Revert immediately and confirm the suite passes again.

What is mutation testing for test quality?▼

Mutation testing deliberately introduces small faults like inverted conditions or removed validations into code to check whether tests detect them. A test that still passes under the fault does not discriminate correct from incorrect behavior.

Why does my test pass even when the bug is present?▼

The assertion may hold under both the fix and the bug, or it may assert implementation trivia instead of observable behavior. Reshape the scenario so the correct answer differs from what the bug would produce.

How do I safely revert mutated code without losing uncommitted work?▼

Copy the file before mutating it and restore from that copy. Never use git checkout or git restore while work is uncommitted, since those discard your changes along with the mutation.

When is mutation testing not practical?▼

Mutation is impractical only after establishing the runtime is genuinely unavailable, not by assumption. In that case, name the smallest fault that should be tried next instead of claiming confidence without evidence.

Related Skills

Explore More Software Engineering Skills→
PROdotesPROdotes

Regression Testing Protocol

Enforce regression testing with pytest by requiring reproducing tests for every bug fix.

Community
Intermediate
bordenetbordenet
8

progressive-code-review-gate

Automate progressive code review before commits and pushes.

Community
Advanced
Ramprasad4121Ramprasad4121
1

audit-prep-assistant

Prepare codebases for security review with static analysis and documentation.

Community
Advanced
nateanatea

mutation-testing

Run mutation analysis to compute mutation score and detect surviving mutants.

Community
Advanced