token-optimizer

You can reduce token count in long prompts and documentation by running Python scripts that apply Strunk's prose rules, outputting compressed text alongside token statistics to improve LLM context efficiency.

TOON serialization is a data conversion method that minimizes context size by converting data objects into the TOON format. It is used to reduce token consumption when passing JSON or YAML data to an LLM.

Yes, the token-optimizer works with JSON and YAML data inputs by compressing and standardizing them for AI input. It processes these formats to minimize context size and improve LLM readability.

To compress a Markdown README, you run the compress.py script with your target file. The script applies prose compression techniques to standardize the document and output token statistics for quick comprehension.

You need tiktoken installed if you want to use optional support for accurate token counting in your prompts. The scripts can output compressed text and statistics, with tiktoken providing the underlying token calculation.

You should avoid prose compression when maximum detail and exact wording are legally or technically required, because applying Strunk's rules to reduce token count intentionally removes verbose phrasing to produce concise text.