sglang-model-gateway

You can route OpenAI-compatible HTTP and gRPC traffic to vLLM and SGLang workers by deploying a Rust gateway that uses policy-driven load balancing and Kubernetes service discovery to manage per-model deployments.

Yes, the Rust model gateway supports cache-aware load balancing alongside random, round_robin, power_of_two, prefix_hash, and manual policies to preserve prefix locality and ensure predictable routing behavior.

Yes, you can operate vLLM and SGLang inference workers in an air-gapped environment by using local model mirrors and requiring local tokenizer and model file snapshots on the workers.

Using a Rust gateway with static worker lists or Kubernetes service discovery is the best way to load balance SGLang traffic, as the gateway does not use HF_ENDPOINT and relies on local model files.

Yes, certain policy-driven routing configurations require local tokenizer and model files on the workers to function correctly, especially when operating offline or in air-gapped mirror setups.