lmcache
Subscription: Developer
Active
Subscription: Developer
A serving engine extension for large language models that reduces time to first token and increases throughput by reusing KV caches across requests, particularly effective for long-context inference scenarios and multi-turn conversations.
pip install lmcache==0.5.6
0.5.6
Linux
0.5.5
Linux
0.4.7
Linux