Consideration for text mutation

[sushraja-msft]

One use case for the proofreader API is going to be proof reading as the user types in text. Which raises the question of should this API expose ways for the developer to indicate that new text was appended so that proof reading can be efficient while the user is typing.

It turns out, this is not actually necessary to be exposed at the API level. User agents can diff the text passed to the proofreader API and infer that new text is purely an append or that there is a large substring match between the old and new text. User agents can then optimize their inference engine to roll back state (rewind KV cache) and process just the new content.

Raising this issue, so as to hear if there are any other developer scenarios that require communicating changes in content to the proofreader API. If not, perhaps it would be good to include in the explainer this perf consideration for implementors to optimize subsequent calls to the proofreader API where only a suffix for the text differs.

Testing with the prompt API, it appears chrome already has this subtle optimization to not recompute KV caches when there are substring matches are found in subsequent prompts. Something we would not have discovered unless we thought of this use case.

[scblaze]

This is an important point. Specifically, the following capabilities/behavior would be valuable for us:

• For performance reasons, if already analyzed text is changed or modified only the portions that are modified need to be reanalyzed. E.g. if I have a five page document and change a word of page 2, hopefully only the sentence with the modified word would be reanalyzed.
• Some sort of stable identifiers for errors. E.g. if I have shown an error to the user and they have "dismissed" it or told me to ignore it, it should be easy for me to filter out the same error if the document is reanalyzed. location indexes are not sufficient for this as they may have shifted due to the user making changes to the document.

These sorts of things can be handled by the API consumer, but they are difficult problems and it would be good if they were built-in.