skills.rest
mims-harvard
mims-harvardOFFICIAL
Explore all skills by this author →
Website𝕏@marinkazitnikGitHub

tooluniverse-variant-predictor-dms-validation

Validates variant-effect predictors against deep mutational scanning data using Mann-Whitney U tests.

1.7k|254|Updated Mar 3, 2025
Search Tags:#ESM#alphamissense#mann-whitney u#variant-effect prediction#protein benchmarking#mavedb#deep mutational scanning
One-click install
npx skills add https://github.com/mims-harvard/ToolUniverse --skill tooluniverse-variant-predictor-dms-validation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tooluniverse-variant-predictor-dms-validation
Source: https://github.com/mims-harvard/ToolUniverse/tree/main/plugins/tooluniverse/skills/tooluniverse-variant-predictor-dms-validation
Command: npx skills add https://github.com/mims-harvard/ToolUniverse --skill tooluniverse-variant-predictor-dms-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, scipy, matplotlib.

What problem does it solve?

Researchers building or selecting variant-effect predictors (AlphaMissense, ESM SAE features, EVE, conservation scores) need statistical evidence that their predictor's scores actually track experimental functional measurements on a specific protein, rather than trusting the predictor blindly.

Core Features & Use Cases

  • Predictor-agnostic benchmarking: Scores any per-variant numeric predictor against DMS effect matrices retrieved from MaveDB, with built-in support for ESM-C SAE drops, AlphaMissense bins, ESM logits, and ESM-2 masked-marginal LLR.
  • Statistical validation pipeline: Stratifies variants into neutral vs disruptive groups, runs one-sided Mann-Whitney U tests, and sweeps stratification thresholds (neutral band, disruptive quantile, top-K) for robustness evidence.
  • Failure detection: Mandatory pre-test sanity gates catch silent NaN failures in predictor computation and verify sign conventions via Spearman correlation before interpreting results.
  • Use Case: A researcher publishing a new SAE-based variant scoring method runs this skill on a KRAS DMS dataset to produce a benchmark figure showing p-values and effect sizes across a parameter sweep for reviewer robustness evidence.

Quick Start

Validate my AlphaMissense predictor against the KRAS DMS dataset urn:mavedb:00000115-a-7 and tell me whether its scores reliably distinguish disruptive from neutral variants.

Frequently Asked Questions about tooluniverse-variant-predictor-dms-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate a variant-effect predictor against DMS data?▼

Retrieve a DMS effect matrix with MaveDB_get_effect_matrix, compute per-variant predictor scores aligned to the same (20 × n_positions) layout, stratify variants into neutral and disruptive groups, then run a one-sided Mann-Whitney U test on the predictor scores between groups.

Which variant predictors can be benchmarked against deep mutational scanning data?▼

Any predictor producing per-variant numeric scores works: ESM-C SAE feature drops, AlphaMissense pathogenicity bins, ESM logits-based scores, ESM-2 masked-marginal log-likelihood ratios, EVE scores, conservation scores, or custom in-house models.

Can I use AlphaMissense scores without an API key for DMS validation?▼

Yes. The AlphaMissense tools proxy a public hegelab.org API requiring no key, returning residue-level categorical bins. For keyless ESM scoring, ESM2_score_missense_variant uses HuggingFace's free inference provider to compute log-likelihood ratios.

Why does my Mann-Whitney U test show no significance for a good predictor?▼

Common causes include a flipped disruptive_tail sign convention, silent NaN failures in predictor computation, or a DMS assay measuring something the predictor was not trained for. Run the mandatory NaN coverage gate and Spearman sign check before interpreting results.

What are the limitations of Mann-Whitney U validation on DMS data?▼

Results are per-protein and per-assay only, so they do not generalize across protein families or assay types. MWU also assumes independence, but variants at the same position are mildly correlated, making p-values slightly optimistic.

How do I compare two variant predictors on the same DMS dataset?▼

Run the validation workflow twice with identical stratification parameters, then compare p-values, median gaps between disruptive and neutral groups, NaN coverage, and robustness across the threshold sweep. The predictor with lower p-values and consistent sweep results is stronger.

Related Skills

Explore More Education & Research Skills→
NeuroAIHubNeuroAIHub
269

dbsnp-database

Query NCBI dbSNP to resolve genetic variants and retrieve clinical significance.

Official
Advanced
NeuroAIHubNeuroAIHub
269

alphagenome-single-variant-analysis

Analyze non-coding variant effects on expression, chromatin, and splicing via AlphaGenome API.

Official
Advanced
biomaps-infrabiomaps-infra

db-alphafold

Retrieve AlphaFold protein structures by UniProt ID and download PDB/mmCIF files.

Official
Intermediate
biomaps-infrabiomaps-infra

db-gnomad

Query gnomAD genetic variant allele frequencies and gene constraint metrics via GraphQL API.

Official
Intermediate