character-encoding-detection

Detects and converts character encodings to UTF-8 using encoding_rs library.

836|66|Updated Feb 3, 2025
One-click install
npx skills add https://github.com/kreuzberg-dev/html-to-markdown --skill character-encoding-detection
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: character-encoding-detection
Source: https://github.com/kreuzberg-dev/html-to-markdown/tree/main/.ai-rulez/domains/html-parsing/skills/character-encoding-detection
Command: npx skills add https://github.com/kreuzberg-dev/html-to-markdown --skill character-encoding-detection

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires encoding_rs.

What problem does it solve?

This Skill ensures that text data is correctly interpreted and converted to the standard UTF-8 encoding, preventing display errors and data corruption.

Core Features & Use Cases

  • BOM Detection: Identifies Byte Order Marks to determine encoding.
  • Meta Tag Parsing: Reads HTML5 charset meta tags for encoding information.
  • Auto-Detection: Falls back to robust auto-detection algorithms when other methods fail.
  • Conversion: Converts various character encodings to UTF-8.
  • Use Case: When processing web page content or text files from diverse sources, this skill guarantees that all text is consistently represented in UTF-8, making it safe for storage and display.

Quick Start

Convert the provided text data to UTF-8 format.

Frequently Asked Questions about character-encoding-detection

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert text with unknown character encoding to UTF-8?

To convert text with unknown character encoding to UTF-8, this Skill uses the encoding_rs library to detect the original charset and transform the data. It handles Byte Order Marks and HTML5 meta tags to ensure accurate conversion into valid UTF-8 output.

What is the best way to detect character encoding from HTML5 web pages?

The best way to detect character encoding from HTML5 web pages is by parsing charset meta tags. This Skill reads HTML5 meta tags directly, falling back to BOM detection and auto-detection algorithms to guarantee the text is correctly converted to UTF-8.

Why does my text display corrupted characters and how do I fix it?

Text displays corrupted characters due to incorrect character set interpretation. You fix it by using this Skill to detect the original character encoding via BOM or auto-detection, converting the text data to standard UTF-8 to prevent display errors and data corruption.

Does encoding_rs support BOM detection for character set conversion?

Yes, encoding_rs supports BOM detection for character set conversion. This Skill leverages encoding_rs to identify Byte Order Marks, validate encoding support, and automatically convert various character sets into valid UTF-8 output.

Can I use auto-detection to convert text files to UTF-8 when the charset is missing?

Yes, you can use auto-detection to convert text files to UTF-8 when the charset is missing. When BOM detection and HTML5 meta tag parsing fail, this Skill falls back to robust auto-detection algorithms to determine the encoding and convert the data.