CohereChatGenerator using command-r model doesn't work with Agent

[vblagoje]

Describe the bug
command-r (Cohere model) fails during tool calling in a Haystack Agent pipeline. Although our test suite passes when evaluating tool use with the live model, actual runs frequently trigger a PipelineRuntimeError due to the model not returning valid tool calls or responses.

Error observed:

haystack.core.errors.PipelineRuntimeError: The following component failed to run:
Component name: 'chat_generator'
Component type: 'CohereChatGenerator'
Error: status_code: 422, body: {'message': 'No valid tool call or response generated'}

To Reproduce
Run a Haystack Agent pipeline using CohereChatGenerator with the command-r model and a valid tool registered. The agent setup completes, but at runtime, the model fails to return a proper tool call.

This works with command-r-plus, and with command-a-03-2025 but fails consistently with command-r.

Describe your environment (please complete the following information):

• OS: macOS 14.4.1
• Haystack version: 2.13.0
• Integration version: cohere-haystack-3.0.0

Notes:
At first I thought this must have been related to MCP as we have tool tests in test suite but I eliminated that possibility by testing an Agent instance with single Tool used in integration tests (fake weather tool) and that doesn't work either so it is likely something related to Agent, tools and command-r combination

[sjrl]

Hey @vblagoje out of curiosity would you call this a bug or just a limitation of the command-r model? It seems like the newer models are able to better handle tool calling, so it's possible command-r just fails in this respect more often.

[vblagoje]

Hey @vblagoje out of curiosity would you call this a bug or just a limitation of the command-r model? It seems like the newer models are able to better handle tool calling, so it's possible command-r just fails in this respect more often.

@sjrl I'm suspecting (but we need to verify) command-r is using an old function/tool calling definitions and can't recognize these new ones coming from MCP, or the OpenAI format that we use by default.

[vblagoje]

No, wrong, I forgot to update CohereChatGenerator to handle ToolSet @sjrl - that could be the cause why command-r is not working.

[vblagoje]

@sjrl here is what happened - https://docs.cohere.com/v2/docs/command-r#model-details lists command-r models. In September 24, Cohere migrated to v2 API which changed tool calling format among many other things. They didn't move the command-r marker to a new model compatible with v2 command-r-08-2024 and for some reason this old command-r doesn't digest tool definitions well. The new command-r-08-2024 works perfectly fine for our tooling.

Given these revelations I propose that in #1691 we update the default model to command-r-08-2024 and leave a warning that command-r will not work with our tooling architecture. Newer generation models command-r-plus and command-a work perfectly fine. cc @julian-risch

[sjrl]

@vblagoje that sounds good to me!

[vblagoje]

Fixed with #1691