Extract code blocks only after Code marker

[albertvillanova]

Extract code blocks only after Code marker.

Fix #1219.

[albertvillanova]

This change assumes that only one "Code:" marker appears in the model output.

@aymeric-roucher do you think this is a sensible assumption? Alternative assumptions?

[aymeric-roucher]

I think it's more reasonable to assume that the model will often forget to put the "Code:" header. To solve #1219, it would be more adapted IMO to just enforce in the regex "header is py or python" rather than "Code: has been generated before the curent code blob"

[albertvillanova]

This PR does not enforce the presence of the "Code:" marker: it can handle model output with or without "Code:" marker.

• See test case: https://github.com/huggingface/smolagents/pull/1223/files#diff-33c13e0b177bacd2f02e29bcb8aea5b49e7ce34901fd8f41fefb65defba1bd33R164

Additionally, note that the word "py" or "python" after the triple backtick is always optional.

My question above was:

• Can we assume that the model only outputs one-or-zero "Code:" marker? This is covered in this PR.
• Or could it output multiple "Code:" markers? This is not covered in this PR.

[aymeric-roucher]

To clarify: I know that currently the header py or python is optional (cf the regex).
The goal here is to differentiate markdown code blobs that are not the agent's action (could be for instance the LLM generating a plan in pure markdown and separating it from the rest of its generation with triple backticks) from the code blobs that are the agent's real action in python.

Sometimes LLM generate their action in 2 parts, and forget to put the "Code:" header

This is why, I think it makes more sense to enforce "action code blobs have a mandatory header py or python" (which is not currently the case) than to enforce "action code blobs have the 'Code:' sequence before them" as this PR is currently proposing.

[albertvillanova]

Thanks for the clarification, @aymeric-roucher.

Just a naive question: do you think it could be plausible that the model might generate a py/python code block within the "Thought" section (e.g., as part of reasoning or planning), which should not be parsed as an action code block?

If so, maybe a combined approach would be more solid...

Curious to hear your thoughts on that edge case.

[aymeric-roucher]

@albertvillanova it is possible indeed! But in terms of reducing false positives / false negatives, I think the solution "force the heading with py or python" could have some false positives, but it will greatly reduce false negatives compared to the solution "Take code blobs only after the Code: sequence"