Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and parallelism.
npx skills add https://github.com/ricable/mcai --skill tensorrt-llm-ricable
Please help me install this Agent Skill. Skill: tensorrt-llm Source: https://github.com/ricable/mcai/tree/main/.agents/skills/tensorrt-llm Command: npx skills add https://github.com/ricable/mcai --skill tensorrt-llm-ricable
This Skill helps you deploy large language models on NVIDIA GPUs with much higher throughput and lower latency than standard PyTorch inference, while keeping serving reliable for production workloads.
Use the tensorrt-llm skill to choose an NVIDIA GPU deployment setup, optimize the model with the right quantization and batching strategy, and prepare a production-ready serving configuration.
High-intent search queries and answers about installing and using this skill.
To optimize LLM inference latency on NVIDIA GPUs, apply FP8 or INT4 quantization, enable in-flight batching, paged KV cache, and CUDA graphs. These techniques reduce memory overhead and significantly improve throughput compared to standard PyTorch inference.
The best way to serve a 70B model on multiple H100 GPUs is configuring tensor parallelism and pipeline parallelism. This distributes the model across GPUs to maximize throughput and minimize latency for production workloads.
To expose an OpenAI-compatible API for a quantized model, configure production serving with health checks and metrics. Deploy the optimized TensorRT-LLM configuration using Docker or Kubernetes for real workloads.
Yes, you can use in-flight batching and paged KV cache for production LLM serving. These features batch multiple requests dynamically and manage memory efficiently, ensuring reliable high-throughput deployment on NVIDIA GPUs.
Yes, TensorRT-LLM supports INT4 quantization and chunked context. These features allow you to reduce model size and process long contexts efficiently, lowering deployment costs while maintaining inference speed.
To set up multi-GPU tensor parallelism for large language models, configure the parallelism strategy to span multiple GPUs or nodes. This approach enables high-throughput inference for models exceeding single-GPU memory limits.
arsityOptimize LLM inference throughput and latency on NVIDIA GPUs with TensorRT-LLM.
tadod12Serve LLM inference via OpenAI-compatible endpoints using vLLM.
haJ1tGenerates Dockerfiles, Kubernetes manifests, Helm charts, and service mesh configurations for containerized applications.
informatico-madridOptimizes LLM serving on NVIDIA hardware with batched GPU inference.
