Serve LLMs with RadixAttention prefix caching and structured generation.
npx skills add https://github.com/uv-xiao/pkbllm --skill uv-sglang
Please help me install this Agent Skill. Skill: uv-sglang Source: https://github.com/uv-xiao/pkbllm/tree/main/knowledge/ML/serving/sglang Command: npx skills add https://github.com/uv-xiao/pkbllm --skill uv-sglang
This Skill addresses the performance bottlenecks in serving Large Language Models (LLMs), particularly for repetitive tasks like agentic workflows and structured output generation, by significantly accelerating inference speed and improving throughput.
Launch the SGLang server with a Llama 3-8B model on port 30000.
High-intent search queries and answers about installing and using this skill.
You can speed up LLM serving for agentic workflows by using RadixAttention prefix caching, which reuses KV caches for system prompts and conversation history to drastically reduce latency for subsequent tool calls.
The best way to enforce structured generation is using a serving framework with built-in RadixAttention, which reliably generates JSON, regex-constrained, or grammar-based outputs crucial for agent tool calls and data parsing.
Yes, you can use SGLang with existing applications because it offers an OpenAI-compatible API for seamless integration, supporting various models and hardware for high-performance inference.
RadixAttention prefix caching improves multi-turn conversation latency by automatically storing and reusing the KV cache of shared system prompts and conversation history across turns, avoiding redundant computation.
Yes, you need specific dependencies including sglang, torch, and transformers to run structured generation workflows and launch the high-performance LLM serving server.
zechenzhangAGIPrune large language models using one-shot methods like Wanda and SparseGPT.
sahiixxGenerate embeddings, analyze images, and extract structured JSON from text and documents.
dotruruCache shared KV prompts with radix-tree based RadixAttention for LLM serving.
ihatesea69Serve LLMs with RadixAttention prefix caching and constrained decoding.
