What do I do when I run out of KV cache, but the conversation RequiresInference?

[lostmsu]

I used to do conversation.ShiftLeft(count: (int)this.ContextSize / 2), but just now I got CannotModifyWhileRequiresInferenceException.

So what to do in this case? Inference is impossible due to running out of KV cache space. And shifting is impossible due to this exception.

Can I check the space in advance somehow? Any other suggestions?

[martindevans]

Doing ShiftLeft while you have tokens prompted (i.e. waiting for inference) isn't valid.

e.g. if your sequence is the tokens:

[ A, B, C, D, E, F ]

Next time you prompt, it'll store G at index=6. Now if you do ShiftLeft(3) your sequence is:

[ D, E, F ]

Which means the batched work (still storing G@6) makes no sense! That's the mistake which the CannotModifyWhileRequiresInferenceException exception is protecting you from.

As you've found this is a bit awkward, if you have just one active Conversation you're in an almost unrecoverable situation. Your best option is probably to look at the Conversation.TokenCount property (before prompting) to check if you're about to run out of space.

That's definitely a bit of an API design wart that needs cleaning up! Any ideas that would make it nicer to work with?