vllm-bootstrap

What problem does it solve?

vLLM servers often fail to start or behave incorrectly due to dependency/version mismatches, wrong tensor-parallel settings, tool-calling parser incompatibilities, and GPU memory/OOM issues during model loading and request handling.

Core Features & Use Cases

• Verified dependency matrix: Provides a tested set of package versions (vLLM, transformers, flashinfer-python/cubin, mistral_common) and warns about known-bad combinations.
• Launch configuration guide: Defines a standard OpenAI-compatible server startup command and explains key flags like tensor parallel size (TP), max context length, and GPU memory utilization.
• Failure-to-fix playbook: Covers common boot/runtime errors (DeepGEMM issues, flashinfer cubin mismatch, ReasoningEffort import errors, tool-call parser mismatches, NCCL TP hangs, OOM, port conflicts) with concrete remediation steps.
• Tool-call parser mapping: Maps tool-call parser choices (hermes/mistral/pythonic/etc.) to model families to reduce “tool calls not parsed” failures.
• Verification & monitoring: Includes curl-based smoke tests for /v1/models, chat completions, and tool calling plus guidance for log tailing and GPU metric checks.

Quick Start

Ask an AI to guide you to install the verified dependencies and launch an OpenAI-compatible vLLM server with a chosen tensor-parallel size and tool-call parser that matches your model, then run the /v1/models and chat/tool-calling smoke tests to confirm it is working.