lmcache
Subscription: Developer
Active
Subscription: Developer
A serving engine extension for large language models that reduces time to first token and increases throughput by reusing KV caches across requests, particularly effective for long-context inference scenarios and multi-turn conversations.
pip install lmcache==0.5.5
0.5.5
Linux