token-morph-metrics

What problem does it solve?

This Skill provides comprehensive metrics to evaluate Arabic tokenizers' morphological fidelity and boundary accuracy, facilitating fair comparisons and improvements.

Core Features & Use Cases

• Intrinsic Metrics Computation: Calculates root, pattern, morpheme integrity, clitic separation, and fragmentation ratios.
• Tokenizer Assessment: Supports analyzing various tokenizer architectures, including subword, character, and byte-level models.
• Use Case: Researchers can determine how well a tokenizer preserves Arabic morphology without running downstream tasks or external models.

Quick Start

Run the analysis script with your tokenizer output files to obtain the morphological metric scores and insights.