Serve large language and multimodal models with OpenAI-compatible APIs.
npx skills add https://github.com/quantmind-br/skills --skill sglang-quantmind-br
Please help me install this Agent Skill. Skill: sglang Source: https://github.com/quantmind-br/skills/tree/main/sglang Command: npx skills add https://github.com/quantmind-br/skills --skill sglang-quantmind-br
SGLang provides a self-hosted, high-performance serving framework for large language and multimodal models, enabling organizations to deploy inference at scale with OpenAI-compatible APIs and hardware-aware optimizations.
Launch the SGLang server with a model-path and host/port, then send a request to the /v1 endpoints to begin serving.
High-intent search queries and answers about installing and using this skill.
You can serve large language models with an OpenAI-compatible API in a self-hosted environment by launching a high-performance inference server using SGLang. It exposes standard /v1 endpoints for chat, completions, embeddings, and vision tasks.
The best way to run offline batch inference for multimodal models is using a high-performance serving framework. SGLang supports offline batch processing alongside hardware-aware optimizations for efficient document understanding pipelines.
Yes, you can deploy high-throughput LLM inference across multiple GPUs and large clusters. SGLang provides distributed inference capabilities with hardware-aware optimizations like KV caching and speculative decoding for scalable serving.
Self-hosted LLM serving with SGLang supports both quantization and speculative decoding to maximize performance. These hardware-aware optimizations improve inference throughput and reduce latency across diverse GPU configurations.
To start an inference server for chat completions and embeddings, launch SGLang with a specified model-path and host/port. Once running, send requests directly to the standard OpenAI-compatible /v1 endpoints to begin serving.
evalstateSubmit TRL training jobs on Hugging Face Jobs and convert models to GGUF.
jaredcroxtonDeploy OpenAI-compatible LLM inference servers with vLLM and quantization.
JKhyroServe large language models with OpenAI-compatible endpoints and high-throughput inference.
LynxLabVNServe large language models with OpenAI-compatible APIs and quantization.
