Launch and tune SGLang servers with constrained decoding and RadixAttention.
npx skills add https://github.com/jayll1303/AIEKit --skill sglang-serving
Please help me install this Agent Skill. Skill: sglang-serving Source: https://github.com/jayll1303/AIEKit/tree/main/.kiro/skills/sglang-serving Command: npx skills add https://github.com/jayll1303/AIEKit --skill sglang-serving
Deploying and tuning LLM inference servers that require structured outputs, shared-prefix caching, and multi-GPU scaling is complex and error-prone. This skill consolidates launch patterns, performance tuning, and diagnostics for SGLang Runtime to reliably serve constrained JSON/regex/EBNF outputs, enable RadixAttention prefix caching, and run quantized models in production.
Launch an SGLang server for meta-llama/Llama-3.1-8B-Instruct on port 30000 and verify the /v1/models endpoint returns the model list.
High-intent search queries and answers about installing and using this skill.
Serve LLMs with structured constrained decoding using native json_schema, regex, and EBNF formats to enforce always-valid JSON and grammar-constrained responses. This ensures format-constrained outputs directly during high-throughput inference.
RadixAttention prefix caching improves LLM inference throughput by storing and reusing computations for shared-prefix workloads like long system prompts and few-shot examples. This maximizes throughput when many clients share identical context prefixes.
Configure tensor parallelism for multi-GPU LLM serving by setting tp and dp flags during server launch. This scales inference across GPUs while avoiding NCCL/NVLink pitfalls and OOM errors through recommended mem-fraction tuning and VRAM estimation.
You can serve quantized FP8, AWQ, and GPTQ models with SGLang to reduce memory footprint. The runtime provides specific guidance for quantized model serving scenarios to ensure stable multi-GPU deployment without OOM errors.
LLM inference servers run out of memory or fail to load models due to incorrect mem-fraction sizing or TP requirements. Diagnose OOM errors and model load failures using provided VRAM estimation procedures and chunked prefill tuning recommendations.
Launch an OpenAI-compatible API server for Llama models using sglang.launch_server with model-path and port flags. Verify the /v1/models endpoint returns the model list to confirm successful deployment of the high-throughput runtime.
cryptopafiServe large language models with high-throughput inference for OpenAI-compatible endpoints.
hsliuustc0106Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.
spacecat2002Diagnose hanging SGLang distributed inference jobs by locating blocked NCCL collectives and divergent rank states.
eddieluengServe LLMs via vLLM with PagedAttention and continuous batching.
