A high-throughput and memory-efficient inference and serving engine for large language models. It utilizes PagedAttention to efficiently manage memory while providing state-of-the-art throughput and continuous batching.

vllm

vllm-0.31.0-3.1-cp38-abi3-manylinux_2_39_aarch64.whl

vllm-0.31.0-3.1-cp38-abi3-manylinux_2_39_x86_64.whl

vllm-0.30.0-5.1-cp38-abi3-manylinux_2_39_aarch64.whl

vllm

0.31.0

0.30.0

0.29.0

0.22.0