vllm-mindspore-post-process

Name: vllm-mindspore-post-process
Availability: InStock
Author: MindSpore

Optimize vLLM MindSpore post-processing with penalties, temperature, and Top-K/Top-P filtering.

258|48|Updated Jun 22, 2020

Search Tags:#sampling #inference #vllm #post-processing #penalties #mindspore

Ready to run

npx skills add https://github.com/mindspore-ai/akg --skill vllm-mindspore-post-process

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: vllm-mindspore-post-process
Source: https://github.com/mindspore-ai/akg/tree/main/akg_agents/workspace/.opencode/skills/vllm-mindspore-post-process
Command: npx skills add https://github.com/mindspore-ai/akg --skill vllm-mindspore-post-process

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

后处理阶段对计算结果进行优化处理，包括缓存、并行化、向量化等，提升推理速度与资源利用效率。

Core Features & Use Cases

Penalty optimization: 在后处理阶段应用重复惩罚、频率惩罚和存在惩罚的高效实现，降低生成重复和无关输出的概率。
Temperature and sampling control: 通过温度控制改写概率分布，提升输出多样性与稳定性。
Top-K/Top-P filtering: 实现高效的 Top-K 与 Top-P 过滤，降低候选 token 集合大小，提升推理吞吐。
Performance-oriented workflow: 支持按需计算、范围缩减、非就地操作等优化模式，提升整体性能。

Quick Start

Run the vllm-mindspore-post-process pipeline on your model outputs to apply penalties, temperature scaling, and Top-K/Top-P filtering.

Frequently Asked Questions about vllm-mindspore-post-process

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I apply penalties and temperature scaling during MindSpore vLLM inference?▼

To apply penalties and temperature scaling during MindSpore vLLM inference, you can use post-processing functions like apply_penalties and apply_temperature to modify probability distributions and reduce repetitive outputs.

What is the best way to filter candidate tokens in MindSpore vLLM to improve throughput?▼

The best way to filter candidate tokens in MindSpore vLLM is by applying Top-K and Top-P filtering to reduce the candidate token set size, which directly improves overall inference throughput and latency.

Can I use non-inplace operations for vLLM post-processing to optimize performance?▼

Yes, you can use non-inplace operations for vLLM post-processing, as the workflow supports on-demand computation and range-reduction modes to optimize performance and improve resource efficiency.

Does vLLM post-processing support repetition, frequency, and presence penalties?▼

Yes, vLLM post-processing supports repetition, frequency, and presence penalties through the apply_penalties function, lowering the probability of generating repetitive or irrelevant model outputs.

Why does my MindSpore vLLM model generate repetitive outputs and how can I fix it?▼

MindSpore vLLM models generate repetitive outputs when post-processing lacks penalty configurations, which you can fix by applying configurable penalties and temperature control to stabilize the probability distribution.