pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation

[shuvo881]

When I use Llama3.2. that time I got the error.

Code:

from dataclasses import dataclass

from pydantic import BaseModel, Field

from pydantic_ai import Agent, RunContext
from dotenv import load_dotenv

# load_dotenv()

class DatabaseConn:
    """This is a fake database for example purposes.

    In reality, you'd be connecting to an external database
    (e.g. PostgreSQL) to get information about customers.
    
    """
    

    users = [
        {
            'id': 123,
            'name': 'John',
            'balance': 123.45,
            'status': 'active',
            'card_blocked': False,
        
        },
        {
            'id': 456,
            'name': 'Kawya',
            'balance': 678.90,
            'status': 'active',
            'card_blocked': False,
        },
    ]

    @classmethod
    async def customer_name(cls, *, id: int) -> str | None:
        for user in cls.users:
            if user['id'] == id:
                return user['name']
        raise ValueError('Customer not found')

    @classmethod
    async def customer_balance(cls, *, id: int, include_pending: bool) -> float:
        for user in cls.users:
            if user['id'] == id:
                return user['balance']
        raise ValueError('Customer not found')
    
    @classmethod
    async def block_card(cls, *, id: int) -> None:
        for user in cls.users:
            if user['id'] == id:
                user['card_blocked'] = True
                return
        raise ValueError('Customer not found')


@dataclass
class SupportDependencies:
    customer_id: int
    db: DatabaseConn


class SupportResult(BaseModel):
    support_advice: str = Field(description='Advice returned to the customer')
    block_card: bool = Field(description='Whether to block their')
    risk: int = Field(description='Risk level of query', ge=0, le=10)


support_agent = Agent(
    'ollama:llama3.2',
    deps_type=SupportDependencies,
    result_type=SupportResult,
    system_prompt=(
        'You are a support agent in our bank, give the '
        'customer support and judge the risk level of their query. '
        "Reply using the customer's name."
    ),
)


@support_agent.system_prompt
async def add_customer_name(ctx: RunContext[SupportDependencies]) -> str:
    customer_name = await ctx.deps.db.customer_name(id=ctx.deps.customer_id)
    return f"The customer's name is {customer_name!r}"


@support_agent.tool
async def customer_balance(
    ctx: RunContext[SupportDependencies], include_pending: bool
) -> str:
    """Returns the customer's current account balance."""
    balance = await ctx.deps.db.customer_balance(
        id=ctx.deps.customer_id,
        include_pending=include_pending,
    )
    return f'${balance:.2f}'

@support_agent.tool
async def block_card(ctx: RunContext[SupportDependencies]) -> bool:
    """Block the customer's card."""
    await ctx.deps.db.block_card(id=ctx.deps.customer_id)
    return True


deps = SupportDependencies(customer_id=123, db=DatabaseConn())
result = support_agent.run_sync('What is my balance?', deps=deps)
print(result.data)
"""
support_advice='Hello John, your current account balance, including pending transactions, is $123.45.' block_card=False risk=1
"""

result = support_agent.run_sync('I just lost my card!', deps=deps)
print(result.data.support_advice)
"""
support_advice="I'm sorry to hear that, John. We are temporarily blocking your card to prevent unauthorized transactions." block_card=True risk=8
"""

print(DatabaseConn.users)

[hecksadecimal]

Experiencing similar results in every tool-enabled ollama model that I've tried so far.
Something seems to be going wrong with translating the final answer to the expected result type.

[hecksadecimal]

A bit more experimentation later. Here is a logfire span detailing a failing run. It appears to use my own defined tools correctly, but falls flat when trying to structure it as defined.

class QueryResponse(BaseModel):
    category: str
    index: str
    reply: str

[samuelcolvin]

This "worked" for me, although the response is incorrect:

from pydantic import BaseModel, Field

from pydantic_ai import Agent


class SupportResult(BaseModel):
    support_advice: str = Field(description='Advice returned to the customer')
    block_card: bool = Field(description='Whether to block their')
    risk: int = Field(description='Risk level of query', ge=0, le=10)


support_agent = Agent(
    'ollama:llama3.2',
    result_type=SupportResult,
    system_prompt=(
        'You are a support agent in our bank, give the '
        'customer support and judge the risk level of their query. '
    ),
)

r = support_agent.run_sync('A customer has lost their card, what should I do?')

print(r.data)
#> support_advice='Please contact our bank’s lost and found department immediately at phone number XXX-XXXX to report your card missing. We will guide you through the next steps and provide a new card as soon as possible.' block_card=False risk=1

If you're seeing issues with a model failing to generate a valid response, I'm afraid that's a limitation of the model, there's not much we can do.

You might want to add:

try:
    result = support_agent.run_sync('What is my balance?', deps=deps)
except UnexpectedModelBehavior:
    print(support_agent.last_run_messages)
    raise

So you can see the messages exchanged and understand what went wrong, or use logfire.

[samuelcolvin]

See #242 - looks like Ollama now has dedicated support for structured responses which might help with this.

[firas665]

I got the same error but fixed it by simply instructing Ollama to output its result in JSON format.