mutation-test-suite

Validates test suites by deliberately mutating code and confirming each mutation is caught.

5|1|Updated Aug 18, 2023
One-click install
npx skills add https://github.com/thpoll83/PolyKybdHost --skill mutation-test-suite-thpoll83
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: mutation-test-suite
Source: https://github.com/thpoll83/PolyKybdHost/tree/main/.claude/skills/mutation-test-suite
Command: npx skills add https://github.com/thpoll83/PolyKybdHost --skill mutation-test-suite-thpoll83

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? A test suite that stays green against deliberately broken code proves nothing. This Skill verifies that googletest (make test:<name>) or Python/unittest suites actually detect breakage by applying 4-7 deliberate mutations to the code under test and confirming each one is caught by the intended test. ## Core Features & Use Cases - Mutation sweep for googletest and Python/unittest: Provides ready-to-adapt shell harnesses that apply a mutation, run the suite, record which test failed, and restore the source between mutations. - Fail-open trap detection: Documents the three silent failure modes (ANSI escape codes hiding gtest failure lines, mutations that never applied, and Qt aborting the interpreter before unittest prints a summary) that make a broken harness report every mutation as "still green". - Structured reporting: Produces a mutation-to-test mapping table suitable for a PR body, proving the tests are worth their line count. - Use Case: After writing a new *_test.py suite for a PyQt feature, run three mutations against the changed module and confirm each is caught by its intended test before opening the PR. ## Quick Start Mutation-test the test suite I just wrote by deliberately breaking the code under test several ways and confirming each mutation is caught by the intended test.

Frequently Asked Questions about mutation-test-suite

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify my test suite actually catches bugs?

Use mutation testing: deliberately break the code under test in 4-7 plausible ways, run the suite after each mutation, and confirm each one is caught by the intended test. A mutation that passes uncaught reveals a gap in the suite.

How to mutation-test a Python unittest suite?

Apply each mutation with a Python heredoc using string replace on the source file, run the suite with python -m unittest, and check for a FAIL or ERROR line matching the expected test. Restore the file from a pre-mutation backup between mutations and verify the mutation applied with diff.

Why does my mutation testing sweep report every mutation as still green?

A unanimously clean sweep usually means the harness is broken, not the suite. Common causes are ANSI escape codes hiding gtest failure lines from your grep, a sed or perl pattern that never matched so no mutation applied, or a Qt abort killing the interpreter before unittest prints its summary.

Can I use git diff to check whether a mutation was applied?

No. You typically mutation-test code you just wrote and have not committed, so the file is already dirty against HEAD and git diff always reports a change. Compare against a pre-mutation backup copy with diff or cmp instead.

When should I not use mutation testing?

Do not use it to find bugs in the code itself, since that is what the tests are for, and do not use it for firmware behavior on hardware. It is meant for validating a suite right after writing it or when reviewing tests that have never failed.