The vLLM Production Stack project provides a reference implementation on how to build an inference stack on top of vLLM with LMCache. It allows you to scale from single vLLM instance to distributed vLLM deployment without changing any application code, monitor the metrics through a web dashboard, and enjoy the performance benefits brought by request routing and KV cache offloading.

vLLM Production Stack

The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components. The purpose of this project is to simplify and automate the configuration of a Prometheus based monitoring stack for Kubernetes clusters.