skills.rest
maziyarpanahi
maziyarpanahiCOMMUNITY
Explore all skills by this author →
Website𝕏@maziyarpanahiGitHub

evaluating-with-leakage-gates

Evaluate de-identification and clinical NER models for residual PHI leakage before release.

5.0k|615|Updated Oct 4, 2025
Search Tags:#compliance#calibration#model evaluation#de-identification#release gates#clinical ner#phi leakage
One-click install
npx skills add https://github.com/maziyarpanahi/openmed --skill evaluating-with-leakage-gates
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-with-leakage-gates
Source: https://github.com/maziyarpanahi/openmed/tree/main/skills/evaluating-with-leakage-gates
Command: npx skills add https://github.com/maziyarpanahi/openmed --skill evaluating-with-leakage-gates

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill prevents unsafe de-identification and clinical NER models from being released when they leak protected health information, miss required identifier recall floors, or fail performance and integrity checks.

Core Features & Use Cases

  • Leakage-First Evaluation: Assess candidate models against release gates G1a through G8, prioritizing residual PHI leakage over aggregate F1.
  • Release Decisions: Produce a fail-closed RELEASABLE or QUARANTINED decision with checks for critical leakage, recall floors, calibration, latency, quantization, regression, and span integrity.
  • Governance Evidence: Generate signed, reproducible gate reports without including raw PHI, and support subgroup fairness audits and model-card handoffs.
  • Use Case: Evaluate a newly quantized clinical de-identification model on a synthetic golden suite, verify that critical identifiers have zero residual leakage, and quarantine it automatically if any gate fails.

Quick Start

Use this skill to run the OpenMed evaluation harness on a synthetic golden suite and determine whether the candidate model is RELEASABLE or QUARANTINED.

Frequently Asked Questions about evaluating-with-leakage-gates

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check clinical de-identification models for PHI leakage before release?▼

To check clinical de-identification models for PHI leakage before release, evaluate them against leakage-first release gates using synthetic golden suites to verify zero residual PHI and fail-closed quarantining.

What is a leakage-first evaluation gate for clinical NER models?▼

A leakage-first evaluation gate assesses clinical NER models by prioritizing residual PHI leakage detection over aggregate F1 scores, applying critical identifier checks, recall floors, and span integrity tests.

How do I generate reproducible governance reports without exposing raw PHI?▼

You can generate signed reproducible governance reports without exposing raw PHI by running an evaluation harness on synthetic golden suites and outputting signed reports with identity metadata and calibration artifacts.

Can I automatically quarantine a de-identification model if it fails recall floors?▼

Yes, you can automatically quarantine a de-identification model by applying fail-closed release decisions that check critical leakage, recall floors, calibration, latency, quantization, and span integrity.

What are the limitations of using aggregate F1 scores for de-identification release decisions?▼

Aggregate F1 scores obscure critical PHI leakage risks, whereas leakage-first gates enforce zero residual leakage on critical identifiers, recall floors, and span integrity to ensure safe fail-closed decisions.

Related Skills

Explore More Legal & Compliance Skills→
maziyarpanahimaziyarpanahi
5.0k

auditing-subgroup-fairness

Audit demographic subgroup leakage and recall in clinical NER models.

Community
Advanced
cristoslccristoslc
2

swain-test

Run two-phase integration and smoke tests via swain-test.sh outputs.

Community
Intermediate
maziyarpanahimaziyarpanahi
5.0k

authoring-model-cards

Convert OpenMed clinical evaluation reports into governance-ready model cards.

Community
Advanced
maziyarpanahimaziyarpanahi
5.0k

gating-deid-leakage

Fail CI when PHI recall drops below approved thresholds or critical identifiers leak.

Community
Advanced