Skip to content

Commit 8ea8573

Browse files
.Net: Fix default value for max_tokens (#2743)
### Motivation and Context #### ⚠️ Breaking change. When `Kernel.InvokeSemanticFunctionAsync` is used, `max_tokens` is set to `256` by default. However if you call `Kernel.CreateSemanticFunction`, `max_tokens` is set to `null` by default so the model default value is used. The reason we are setting `max_tokens` to 256 is because OpenAI text completion models have a default of 16. However OpenAI chat completion models have a default of infinity. So the current default makes sense for text completions but not for chat completions. To fix this, the default value of `max_tokens` will always be set to `null`. #### Troubleshooting Enable informational logging and check for token usage logs e.g., ``` Action: GetCompletionsAsync. Azure OpenAI Deployment Name: text-davinci-003. Prompt tokens: 14. Completion tokens: 16. Total tokens: 30. ``` #### Here's the documentation for OpenAI: https://platform.openai.com/docs/api-reference/completions/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to 16` The maximum number of [tokens](https://platform.openai.com/tokenizer) to generate in the completion. https://platform.openai.com/docs/api-reference/chat/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to inf` The total length of input tokens and generated tokens is limited by the model's context length. Resolves: #2738 ### Description The property `max_tokens` is an optional so we should not set a default value in SK. This change makes SK consistent in this approach. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
1 parent ee68b65 commit 8ea8573

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

dotnet/src/SemanticKernel/SkillDefinition/InlineFunctionsDefinitionExtension.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,7 @@ public static Task<SKContext> InvokeSemanticFunctionAsync(
134134
string? functionName = null,
135135
string? skillName = null,
136136
string? description = null,
137-
int maxTokens = 256,
137+
int? maxTokens = null,
138138
double temperature = 0,
139139
double topP = 0,
140140
double presencePenalty = 0,

0 commit comments

Comments
 (0)