conformance-loop

Compares Fallow analysis output against competing tools and verified source truth to improve accuracy.

4.4k|154|Updated Mar 17, 2026
One-click install
npx skills add https://github.com/fallow-rs/fallow --skill conformance-loop
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: conformance-loop
Source: https://github.com/fallow-rs/fallow/tree/main/.agents/skills/conformance-loop
Command: npx skills add https://github.com/fallow-rs/fallow --skill conformance-loop

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Static analysis tools can drift from ground truth, producing false positives and false negatives that erode trust. This Skill provides a disciplined loop for measuring and improving Fallow's analysis accuracy against competing tools and manually verified source code.

Core Features & Use Cases

  • Comparative Benchmarking: Run Fallow and competing analyzers with documented equivalent settings across a stable real-world project corpus.
  • Disagreement Classification: Manually verify divergent findings against source and classify each as true positive, false positive, false negative, or model difference.
  • Regression-Guarded Fixes: Implement one general correction at a time with a regression fixture, re-run the full corpus, and retain only net improvements.
  • Use Case: When Fallow reports unused exports differently than another dead-code tool on a monorepo, use this loop to verify which tool is correct, fix the underlying analysis gap, and lock in the improvement with a fixture.

Quick Start

Run the conformance loop to compare Fallow's dead-code findings against competing tools on a stable corpus and classify every disagreement.

Frequently Asked Questions about conformance-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify static analysis findings against ground truth?

Run the analyzer and competing tools with documented equivalent settings on a stable corpus, then manually inspect each disagreement against the source code. Classify every divergence as a true positive, false positive, false negative, or model difference before changing any logic.

How to compare dead-code detection tools on a real codebase?

Select a stable set of real-world projects, run each tool with equivalent configurations, and diff the reported findings. Treat competitor output as a lead rather than ground truth, and confirm disputed findings by reading the actual source.

Can competitor tool output be trusted as ground truth?

No. Competitor output is a lead, not ground truth. Each disagreement must be manually verified against the source code, since differences may reflect model differences rather than actual errors in either tool.

When should a fix for a false positive be kept or reverted?

Implement one general correction with a regression fixture, then re-run the full corpus. Retain the change only if it produces a net improvement across the corpus; revert fixes that trade one false positive for new false negatives.

What are the limitations of corpus-based conformance testing?

Results depend on corpus representativeness and equivalent tool configuration, which can be hard to guarantee. Manual verification also scales poorly, so the loop works best on stable, well-understood project sets.