Deploy large language models across GPU clusters with distributed serving and OpenAI-compatible APIs.
npx skills add https://github.com/jstzwj/ai-infra-plugins --skill vllm-jstzwj
Please help me install this Agent Skill. Skill: vllm Source: https://github.com/jstzwj/ai-infra-plugins/tree/main/plugins/vllm/skills/vllm Command: npx skills add https://github.com/jstzwj/ai-infra-plugins --skill vllm-jstzwj
This Skill provides a comprehensive and efficient solution for deploying and running large language models at scale in production environments.
Launch the vLLM server with a specified model and query it via a REST client to generate responses.
High-intent search queries and answers about installing and using this skill.
Deploy a large language model across a GPU cluster using this Skill to execute optimized inference with minimal latency and maximum throughput via CUDA graphs, quantization, and custom kernels. It supports distributed serving with tensor, pipeline, expert, and data parallelism.
Serve a 70B parameter LLM on a 4-GPU cluster using this Skill's distributed serving capabilities, which support tensor, pipeline, expert, and data parallelism to achieve real-time chat and high-throughput inference.
Yes, LLM inference with this Skill supports multi-modal inputs. It is designed to handle image, audio, and video data alongside text inputs when deploying large language models efficiently across GPU clusters.
Yes, you need CUDA and PyTorch to run distributed LLM serving with this Skill. These dependencies are required to execute large models efficiently and utilize custom kernels and quantization for fast inference.
To query a deployed model using an OpenAI-compatible REST API, launch the vLLM server with a specified model and send requests via a REST client to generate responses. It also supports gRPC and offline batch inference.
anilcan-karaOptimize LLM inference with NVIDIA TensorRT for high throughput and low latency.
peteromalletDeploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.
lxt070409Convert video, audio, images, and text into ASCII art videos using Python and ffmpeg.
jayll1303Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.
