readability-lexical-diversity

Use the textstat library to compute Flesch-Kincaid, Coleman-Liau, Gunning Fog, SMOG, and ARI scores per document, then report the median consensus grade and range. Running all five formulas matters because they can diverge by 2-4 grade levels on the same text.

MTLD is the preferred measure because it shows near-zero correlation with text length (r = -0.02), while raw TTR correlates strongly with length (r = -0.70 to -0.90). MATTR with a fixed 50-token window is a solid alternative.

Raw type-token ratio decreases mechanically as text length increases, so longer texts always appear less diverse regardless of actual vocabulary range. Use MTLD or MATTR, which correct for length, when comparing documents or authors.

Flesch-Kincaid, Coleman-Liau, Gunning Fog, and ARI require at least 100 words, with 200+ words recommended. SMOG requires at least 30 sentences (roughly 600 words); below these thresholds scores are unreliable and should be excluded or flagged.

Convert each measured metric into a target range rather than a point value, such as "FK grade 11-14" or "average 17-22 words per sentence." Readability scores carry roughly +/- 1.5 grade levels of imprecision, so ranges are the honest constraint format.

No, these formulas are calibrated for English and produce invalid grade-level estimates on other languages. Non-English documents should be excluded from scoring or explicitly flagged before analysis.