fullwidth_halfwidth_normalizer

What problem does it solve?

This Skill addresses the challenge of inconsistent character width in Chinese documents, ensuring uniformity and compatibility.

Core Features & Use Cases

• Full-width to Half-width Conversion: Transforms full-width ASCII characters and spaces into their half-width equivalents.
• Unicode Normalization: Optional normalization of Unicode characters to standardize their representation.
• Use Case: Ideal for processing Japanese-Korean-Chinese mixed text where a consistent character width is required, without altering Chinese punctuation or formatting.

Quick Start

Normalize the full-width characters in your dataset by running the skill with the following command: python scripts/fullwidth_halfwidth_normalizer.py --input_path input.csv --output_path output.csv.