Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.
npx skills add https://github.com/AissenLiu/EasyHermes --skill tensorrt-llm-aissenliu
Please help me install this Agent Skill. Skill: tensorrt-llm Source: https://github.com/AissenLiu/EasyHermes/tree/main/hermes-agent/optional-skills/mlops/tensorrt-llm Command: npx skills add https://github.com/AissenLiu/EasyHermes --skill tensorrt-llm-aissenliu
TensorRT-LLM optimizes large language model inference on NVIDIA GPUs, delivering higher throughput and lower latency for production deployments.
Install tensorrt_llm and start trtllm-serve with a model to begin serving high-performance LLMs.
High-intent search queries and answers about installing and using this skill.
Deploy TensorRT-LLM to optimize LLM inference on NVIDIA GPUs, achieving higher throughput and lower latency. It applies FP8/INT4 quantization and in-flight batching for production deployment.
Yes, multi-GPU and multi-node LLM serving is supported via tensor, pipeline, and expert parallelism. This scales inference across multiple CUDA-capable GPUs for higher throughput.
Yes, TensorRT-LLM requires CUDA-capable GPUs, a Python environment, and the tensorrt_llm package to run the server and APIs as documented in the references.
Install the tensorrt_llm package and start trtllm-serve with a model. This provides production-ready serving with OpenAI-compatible APIs, health checks, and Prometheus metrics.
Yes, TensorRT-LLM includes Prometheus metrics for monitoring production deployments. It also provides health checks alongside its OpenAI-compatible API serving.
anilcan-karaOptimize LLM inference with NVIDIA TensorRT for high throughput and low latency.
akillnessConfigure Prometheus metrics, structured logging, alert rules, and Grafana dashboards.
OpenLAIROptimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.
dawsonblockOptimize large language model inference with NVIDIA TensorRT-LLM on multi-GPU servers.
