Increase prefix cache memory space

[M0rpheus-0]

Hello!

Is there a way to increase the available memory vLLM uses to store prefixes of past prompts, so that it can cache more prompts?
More so, could we share/allocate some of the CPU RAM to do so?

I am working with 2xH100, and ample RAM to spare.

If not can I at least force a prompt to stay cached?

Thank you!

[gaby]

I have the same use-case.

@M0rpheus-0 They recently moved discussions to a forum, I dont know why. So you wont get an answer here

[M0rpheus-0]

Oh, I see. That's good to know, thanks for the heads up!
Is it this "https://discuss.vllm.ai/"?

Also, I found mentions of LMCache which seems to aim at solving this problem.
It's even mentioned in vLLM's docs.
Their github states that they support an older version of vLLM though.

Do you know anything about this?

[gaby]

Yes, that's the site. I didnt know about LMCache, I will check it out.