vllm-router
Subscription: Developer
Active
Subscription: Developer
A high-performance Rust-based load balancer designed for vLLM inference servers, supporting multiple routing algorithms and prefill-decode disaggregation to efficiently distribute requests across GPU workers for large language model serving.
pip install vllm-router==0.1.15
0.1.15
Linux