feat: default max output tokens for autocomplete #5789
Conversation
✅ Deploy Preview for continuedev canceled.
|
dosubot
bot
added
the
size:S
| @@ -73,6 +74,11 @@ export class CompletionProvider { | |||
| llm.completionOptions.temperature = 0.01; | |||
| } | |||
|
|
|||
| // (DOES NOT WORK) llm.complettionOptions.maxTokens is already populated - need to detect if llm not have maxTokens already set | |||
There was a problem hiding this comment.
need some help here. i want to know if maxTokens in user's config was already set. However, BaseLLM sets the completion option's maxTokens.
There was a problem hiding this comment.
Hey @uinstinct , sorry for the delay in reviewing this!
I'm going to close this out because I think there's a good chance we create a new autocompleteOptions in the near term. Assuming we do that, we can just directly add a new maxTokens property there.
| @@ -195,6 +196,12 @@ export abstract class BaseLLM implements ILLM { | |||
| const templateType = | |||
| options.template ?? autodetectTemplateType(options.model); | |||
|
|
|||
| // if model has a single role - autocomplete, then use a smaller default maxTokens | |||
| const defaultMaxTokens = | |||
| options.roles?.length === 1 && options.roles.at(0) === "autocomplete" | |||
There was a problem hiding this comment.
This solution makes sense but there are some models such as codestral that can be used for both autocomplete and chat.
We are considering creating a top-level autocompleteOptions, similar to chatOptions and embedOptions that already exist. If we go that route than we can just create an autocompleteOptions.maxTokens.
Description
Adds the ability to have a lower maximum output tokens for auto completion by default.
Closes #5449
References #4448 (comment)
References #3994 (comment)
Checklist
Screenshots
[ For visual changes, include screenshots. Screen recordings are particularly helpful, and appreciated! ]
Tests
[ What tests were added or updated to ensure the changes work as expected? ]
Summary by cubic
Set a lower default max output tokens limit (256) for autocomplete to reduce unnecessary token usage.