auditing-subgroup-fairness

To audit clinical NLP models for demographic subgroup bias, apply the audit to OpenMed models using group-tagged synthetic fixtures across sex, age bands, and race or ethnicity surrogates to identify disproportionate leakage or recall failures.

Subgroup leakage analysis in de-identification models measures PHI leakage and recall for synthetic demographic groups such as sex, age band, and race or ethnicity, revealing hidden fairness failures that aggregate metrics often miss.

Detect recall disparity in clinical NER by running an audit that calculates per-group leakage disparity, identifies the worst-performing demographic group, and reports span counts and character totals for context.

You can generate model card fairness reports for clinical AI governance reviews by producing JSON-ready fairness results that document per-group leakage, recall, disparity, worst-group performance, and missing subgroup coverage.

Running a subgroup fairness audit on OpenMed models requires a group-tagged gold corpus and uses the openmed.eval.fairness_report module to summarize per-group recall, leakage, disparity, and data gaps.

Aggregate de-identification performance hides subgroup fairness failures because overall metrics mask weaker clinical NER performance and higher PHI leakage experienced by specific demographic subgroups, requiring targeted disparity detection.