default maxTokens setting for autocomplete

[ferenci84]

Description

Default maxTokens at 256 for autocomplete if there is no overriding user setting for the model. Added autoCompleteMaxTokens

Change from this comment:
#3994 (comment)

Checklist

• The relevant docs, if any, have been updated or created - let's create a separate issue for this once the PR is approved and autoCompleteMaxTokens setting is kept in the completion options.
• The relevant tests, if any, have been updated or created - no relevant tests as far as I know

Testing instructions

I tested by directly putting a log message into the Ollama._streamFim() method.

[netlify]

✅ Deploy Preview for continuedev ready!

Name	Link
🔨 Latest commit	9d26892
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67d73d7a954f600008b5f915
😎 Deploy Preview	https://deploy-preview-4448--continuedev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[sestinj]

@ferenci84 rather than adding even more options to the config, I would much rather allow users to set maxTokens in the completionOptions section of their config for the model

[ferenci84]

The main part of this modification is not the new config option, but the ability to set different default for autocomplete (possibly for individual providers). autocompleteMaxTokens is a technical thing (to let us set default maxTokens for autocomplete for the model without knowing whether that model will be used for autocomplete or something different; and it also allows providers to set maxTokens setting specifically for autocomplete use), the question is whether to document or keep it hidden from users, that's why I didn't add it to the documentation without knowing whether the core team want users being able to tinker with this option or just keep it hidden and/or unavailable (there is an option for both, see my next comment).

[ferenci84]

@sestinj Please look at this: these are possible places to set defaults for maxTokens for autocomplete:

This is setting for global default:

There may also be defaults for individual models:

This is how the final maxTokens is set when making the request. As you can see, maxTokens user setting will always take precedence, the user don't even have to know about the extra key in the BaseCompletionOptions type:

There may be a misunderstanding, I think it's better if the additional key exist just in the type, but not open to users (i.e. hidden, not documented), however they should be used for setting individual defaults for providers, there may be fast and possibly more capable providers that can output more lines of completion, while slower, or those that tend to go into repetition, should be limited.

I believe that adding this key to the BaseCompletionOptions type is a good way to keep it simple for us, developers. If you want to keep it simple for users too, and limit their options, there is a way for us to ignore this additional setting that comes from the user config:

Any way, please let me know if you have an idea to make it better.

[halfline]

out of curiosity, do you know how many tokens on average you're using for autocomplete? Autocomplete pulls in snippets from various parts of the code and it can get kind of lengthy (especially with the hole filler models). on my system, just doing a completion in the quickstart tutorial uses north of 800 tokens.

[ferenci84]

@halfline I do not have such stats. You are talking about the contextLength setting. This modification is about maxTokens that is the limit on the output tokens.

[ferenci84]

@halfline I do not have such stats. You are talking about the contextLength setting. This modification is about maxTokens that is the limit on the output tokens.

[RomneyDa]

@ferenci84 let's hold on changes since not sure about this parameter yet, but some notes:

• Would need to add YAML support (add to defaultCompletionOptions in YAML schema). See previous JSON-only PR and resulting issue
• It seems like with this implementation maxAutocompleteTokens is redundant of maxTokens, since maxTokens will be ignored if maxAutocompleteTokens is present. E.g. it's not detecting if the model is currently being used for autocomplete so isn't different than just using maxTokens (correct me if I'm wrong?)
• The default should probably live in BaseLLM or constants.js

[ferenci84]

• E.g. it's not detecting if the model is currently being used for autocomplete so isn't different than just using maxTokens (correct me if I'm wrong?)

@RomneyDa, maxAutocompleteTokens is only used when the model is used for autocomplete. See that the maxAutocompleteTokens value is used only in the provideInlineCompletionItems function.

The bottom line: This modification is NOT to add a new parameter, but to keep the default maxToken parameter low, in the following conditions:

1. The user doesn't set maxTokens explicitly
   AND
2. The model is used for autocomplete

I was asked to submit a PR for this issue:
#3994 (comment)

The reason to add a new key to the BaseCompletionOptions, and store the default there, is to avoid ovecomplicating things with new types. It can be a different decision whether it's good idea to add a new parameter or not (I would vote NOT), it's not part of this modification, so json or yaml config schema shouldn't be changed.

The default should probably live in BaseLLM or constants.js

I agree. It's possible to add a default per provider. Just as we have default cls.defaultOptions?.completionOptions?.maxTokens, we also have default cls.defaultOptions?.completionOptions?.autoCompleteMaxTokens, except that I added 256 fallback if it's not set (as it will not be set for most providers anyway). We can move the 256 default to any other place.

Please let me know if we are on the same page about the other two questions before I make any further modifications.