[BUG] Tool call results are not properly de-serialized.

[joakiti]

Library name and version

Azure.AI.OpenAI 2.2.0

Describe the bug

According to documentation on this page: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling

We should always append response tool calls to the message history for the OpenAI completions:

                
messages.append({
                    "tool_call_id": tool_call.id,
                    "role": "tool",
                    "name": "get_current_time",
                    "content": time_response,
                })

If we go to this example here, the code clearly expects you to just use the competions object to create a new assistant message:

// Purely for convenience and clarity, this standalone local method handles tool call responses.
string GetToolCallContent(ChatToolCall toolCall)
{
    if (toolCall.FunctionName == getCurrentWeatherTool.FunctionName)
    {
        // Validate arguments before using them; it's not always guaranteed to be valid JSON!
        try
        {
            using JsonDocument argumentsDocument = JsonDocument.Parse(toolCall.FunctionArguments);
            if (!argumentsDocument.RootElement.TryGetProperty("location", out JsonElement locationElement))
            {
                // Handle missing required "location" argument
            }
            else
            {
                string location = locationElement.GetString();
                if (argumentsDocument.RootElement.TryGetProperty("unit", out JsonElement unitElement))
                {
                    return GetCurrentWeather(location, unitElement.GetString());
                }
                else
                {
                    return GetCurrentWeather(location);
                }
            }
        }
        catch (JsonException)
        {
            // Handle the JsonException (bad arguments) here
        }
    }
    // Handle unexpected tool calls
    throw new NotImplementedException();
}

if (completion.FinishReason == ChatFinishReason.ToolCalls)
{
    // WARNING WARNING, THIS IS WHERE IT GOES WRONG
    // Add a new assistant message to the conversation history that includes the tool calls
    conversationMessages.Add(new AssistantChatMessage(completion));

    foreach (ChatToolCall toolCall in completion.ToolCalls)
    {
        conversationMessages.Add(new ToolChatMessage(toolCall.Id, GetToolCallContent(toolCall)));
    }

    // Now make a new request with all the messages thus far, including the original
}

Expected behavior

The expected behaviour is that content from responses is serialized into the content of the response, so that it adheres to the expected string content (tool call, id, args).

To make the above example work, you have to hack it, using something like:

completion.Content.Add(ChatMessageContentPart.CreateTextPart("just an example"));

Otherwise, you get a 400 bad request.

Actual behavior

You get a 400 bad request, using the official example.

Reproduction Steps

Go through the example of the official docs here

Environment

Ran it on 2.2.1 version and 2.2.2-beta-4. Neither serializes the tool calls to the content of the competions object, meaning

messages.Add(new AssistantChatMessage(completion));

Doesn't actually create a correct value for the interface.

[joakiti]

If I get the time, I will create a PR, but basically just have to add this to de-serialization:

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @ralph-msft @trrwilson.