skills.rest
lukefitzg
lukefitzgCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tinysocs-detection-validation-toolkit

Validates TinySocs detection rules end-to-end via xUnit tests and Atomic Red Team harness runs.

Updated Oct 12, 2025
Search Tags:#validation#xunit#opensearch#detection rules#siem#windows security#atomic red team
One-click install
npx skills add https://github.com/lukefitzg/tinysocs --skill tinysocs-detection-validation-toolkit-lukefitzg
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tinysocs-detection-validation-toolkit
Source: https://github.com/lukefitzg/tinysocs/tree/main/.claude/skills/tinysocs-detection-validation-toolkit
Command: npx skills add https://github.com/lukefitzg/tinysocs --skill tinysocs-detection-validation-toolkit-lukefitzg

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Proving that a SIEM detection rule actually fires under a real attack is hard: a rule can pass unit tests yet never trigger end-to-end because of wrong field_match filters, thresholds a test never clears, or missing audit policy on the target machine. This Skill provides the step-by-step recipe for taking one TinySocs detection rule from candidate to harness-validated, with honest pass/fail accounting. ## Core Features & Use Cases - Honest rule authoring: Write field_match filters that match the rule's description, avoiding rules that silently match every event on a channel. - Atomic test authoring: Create tests/atomic-tests.yaml entries with fallback PowerShell commands that clear every expected rule's threshold with margin, plus correct requires/prefer_fallback/timeout_seconds settings. - xUnit fire/silent pairs: Add DetectionEngineTests.cs facts proving a rule fires on-target and stays silent off-target, using the real shipped rules.yml. - Use Case: When asked to "validate rule TS-071" or "why did TS-002 MISS", follow the recipe to stage a build to the Windows VM, run Test-AtomicDetection.ps1 scoped to one technique, and read DETECTED/MISSED/SKIP/ERROR results without mistaking a timeout race for a rule defect. ## Quick Start Ask the assistant to validate detection rule TS-071 end-to-end, from writing the atomic test entry through running the harness on the Windows VM and interpreting the result.

Frequently Asked Questions about tinysocs-detection-validation-toolkit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate a SIEM detection rule end-to-end?▼

Write an honest field_match in rules.yml, add an xUnit fire/silent pair in DetectionEngineTests.cs, author an atomic-tests.yaml entry with a fallback command that clears the rule's threshold with margin, then run Test-AtomicDetection.ps1 scoped to that technique on a Windows VM and read the DETECTED/MISSED/SKIP/ERROR result.

How do I write an Atomic Red Team fallback command that actually triggers a threshold rule?▼

The fallback must clear the threshold of every rule in expected_rules, grouped by the rule's group_by key, with margin rather than exactly at the threshold. For example, a rule grouping by IpAddress at threshold 20 needs 20+ events from one IP; 18 attempts never fires it.

Why did my detection rule MISS in the atomic test harness?▼

Check whether duration_seconds is close to timeout_seconds before concluding the rule is broken; a MISS near the timeout is usually an OpenSearch ingestion race, not a rule defect. Bump timeout_seconds (300 is the safe default) and re-run before writing a postmortem.

What is the difference between xUnit tests and the Atomic Red Team harness for detection rules?▼

xUnit tests prove the rule logic fires on a synthetic matching event loaded from the real rules.yml, with no Windows or OpenSearch needed. The Atomic Red Team harness proves the rule fires end-to-end from a real attack through the Windows event log, agent, and OpenSearch; both are required for harness-validated status.

Why does a test SKIP instead of running in the atomic detection harness?▼

SKIP means the test never ran, either because pilot_status is deferred (the rule is intentionally disabled) or a requires gate failed, such as tamper_protection_disabled, fim_module, domain_controller, or defender_rtp_disabled. A typo in the requires value also silently produces a SKIP, so check spelling against the four accepted values.

When should I not use this detection validation workflow?▼

Do not use it for normalizing raw runs into results JSON, rebuilding the public validation dashboard, or deciding whether a rule should exist at all; those belong to the publication-campaign and research-methodology workflows. Any change that flips a rule's enabled state must also go through change control first.

Related Skills

Explore More Software Engineering Skills→
augustoheissaugustoheiss

agency-threat-detection-engineer

Detect attacker behaviors in telemetry and generate tested Sigma-based SIEM rules.

Community
Advanced
albumentations-teamalbumentations-team
95

benchmark-results-publisher

Convert benchmark result JSONs into documentation-ready tables and figures.

Official
Intermediate
zkksdkzkksdk

safe-self-mod

Validate core file changes in an isolated Clone-01 before applying them.

Community
Advanced
AlphaBitCoreAlphaBitCore
21

test-cursor-adapter

Validate Cursor adapter protobuf normalization through a synthetic GetChatRequest proxy flow.

Official
Advanced