A high-throughput and memory-efficient inference and serving engine for large language models. It utilizes PagedAttention to efficiently manage memory while providing state-of-the-art throughput and continuous batching.

vllm

vllm-0.32.0-1.1-cp38-abi3-manylinux_2_39_aarch64.whl

vllm-0.32.0-1.1-cp38-abi3-manylinux_2_39_x86_64.whl

vllm-0.31.0-3.1-cp38-abi3-manylinux_2_39_aarch64.whl

vllm-0.31.0-3.1-cp38-abi3-manylinux_2_39_x86_64.whl

vllm-0.30.0-5.1-cp38-abi3-manylinux_2_39_aarch64.whl

vllm-0.30.0-5.1-cp38-abi3-manylinux_2_39_x86_64.whl

vllm-0.29.0-4.1-cp38-abi3-manylinux_2_38_aarch64.whl

vllm-0.29.0-4.1-cp38-abi3-manylinux_2_38_x86_64.whl

vllm-0.22.0-4.1-cp38-abi3-manylinux_2_38_aarch64.whl

vllm-0.22.0-4.1-cp38-abi3-manylinux_2_38_x86_64.whl