Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: AgentController accessed before initialization when using S3FileStore #7632

Open
1 task done
zachmostowsky opened this issue Apr 1, 2025 · 2 comments
Open
1 task done

[Bug]: AgentController accessed before initialization when using S3FileStore #7632

zachmostowsky opened this issue Apr 1, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@zachmostowsky
Copy link

zachmostowsky commented Apr 1, 2025 •
edited
Loading

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Describe the bug and reproduction steps

When using the S3FileStore and LocalFileStore classes for session management, it appears that the AgentController._handle_message_action() method is invoked before the class is fully initialized resulting in the below error:

AttributeError: 'AgentController' object has no attribute '_initial_max_iterations'

It is important to note that this only happens during the 2nd request. The first one (when the session is null) works fine. Once the session is not null the error begins to happen. MemoryFileStore does not produce this bug.

From the stack trace (attached), it looks like calls to retrieve session files are happening after the error is already raised.

To note, I am installing OpenHands as a Python package inside my larger application.

Edit: Updated to include that this bug happens for the LocalFileStore class as well.

OpenHands Installation

Other

OpenHands Version

0.29.1 and 0.30.1 both show the issue

Operating System

Linux

Logs, Errors, Screenshots, and Additional Context

Pseudo Implementation:

def execute_openhands(
    cancel_event: Event,
    session_id: str,
    prompt: str,
    runtime_url: str,
    runtime_api_key: str | None,
    config: OpenHandsConfig,
) -> Generator[OpenHandsResult | OpenHandsUsageReport, None, None]:
    async_generator = execute_openhands_async(
        cancel_event=cancel_event,
        session_id=session_id,
        prompt=prompt,
        runtime_url=runtime_url,
        runtime_api_key=runtime_api_key,
        config=config,
    )

    loop = asyncio.new_event_loop()
    try:
        for obj in _iter_over_async(async_generator, loop):
            if isinstance(obj, OpenHandsResult) or isinstance(obj, OpenHandsUsageReport):
                yield obj
            else:
                raise RuntimeError(f"Unexpected object type: {type(obj)}")
    finally:
        loop.close()


async def execute_openhands_async(
    cancel_event: Event,
    session_id: str,
    prompt: str,
    runtime_url: str,
    runtime_api_key: str | None,
    config: OpenHandsConfig,
) -> AsyncGenerator[OpenHandsUsageReport | OpenHandsResult, None]:
    metrics = OpenHandsUsageReport()


    oh_config = get_oh_config(config)

    file_store = S3FileStore(bucket_name="my-log-bucket")

    llm_config = oh_config.get_llm_config(oh_config.default_agent)
    llm = UsageLLM(config=llm_config, usage_metrics=metrics)

    agent_config = oh_config.get_agent_config(oh_config.default_agent)
    agent = CodeActAgent(
        llm=llm,
        config=agent_config,
    )

    event_stream = EventStream(session_id, file_store)

    memory = Memory(
        event_stream=event_stream,
        sid=event_stream.sid,
    )

    runtime = StaticRuntime(
        url=runtime_url,
        api_key=runtime_api_key,
        config=oh_config,
        event_stream=event_stream,
        sid=event_stream.sid,
        plugins=agent.sandbox_plugins,
        env_vars=None,
        status_callback=None,
        attach_to_existing=True,
    )

    user_action = MessageAction(
        content=prompt,
        image_urls=None,
        wait_for_response=False,
        security_risk=None,
    )

    event_stream.add_event(user_action, EventSource.USER)

    # Defines the end states; where the controller should stop executing
    end_states = [
        AgentState.FINISHED,
        AgentState.REJECTED,
        AgentState.ERROR,
        AgentState.PAUSED,
        AgentState.STOPPED,
        AgentState.AWAITING_USER_INPUT,
        AgentState.AWAITING_USER_CONFIRMATION,
    ]

    agent_controller = AgentController(
        agent=agent,
        max_iterations=oh_config.max_iterations,
        max_budget_per_task=oh_config.max_budget_per_task,
        agent_to_llm_config=oh_config.get_agent_to_llm_config_map(),
        event_stream=event_stream,
        headless_mode=True,
    )

    # This executes the OpenHands controller until it reaches one of the end states
    async for _ in run_agent_until_done_or_cancelled(
        controller=agent_controller,
        runtime=runtime,
        memory=memory,
        end_states=end_states,
        metrics=metrics,
        cancel_event=cancel_event,
    ):
        yield metrics

    state = agent_controller.get_state()

    runtime.close()

    last_agent_message = state.get_last_agent_message()
    last_agent_message_content = last_agent_message.content if last_agent_message is not None else None

    if state.agent_state == AgentState.FINISHED:
        yield OpenHandsResult(
            status=ExecutionStatus.SUCCESS,
            message=last_agent_message_content,
            usage=metrics,
        )
    elif state.agent_state in {AgentState.ERROR, AgentState.STOPPED}:
        logger.debug(
            "AI Developer execution failed", last_agent_message=last_agent_message, state_error=state.last_error
        )
        yield OpenHandsResult(
            status=ExecutionStatus.FAILURE,
            message=last_agent_message_content,
            err=retrieve_error(state.last_error),
            usage=metrics,
        )
    elif state.agent_state in {
        AgentState.REJECTED,
        AgentState.AWAITING_USER_INPUT,
        AgentState.AWAITING_USER_CONFIRMATION,
    }:
        yield OpenHandsResult(
            status=ExecutionStatus.USER_ACTION_REQUIRED,
            message=last_agent_message_content,
            usage=metrics,
        )
    else:
        raise RuntimeError(f"Unexpected agent state: {state.agent_state}")

Stack Trace:

backend-1         |   File "/usr/local/lib/python3.12/threading.py", line 1032, in _bootstrap
backend-1         |     self._bootstrap_inner()
backend-1         |   File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
backend-1         |     self.run()
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/threading/__init__.py", line 129, in __wrap_threading_run
backend-1         |     return call_wrapped(*args, **kwargs)
backend-1         |   File "/usr/local/lib/python3.12/threading.py", line 1012, in run
backend-1         |     self._target(*self._args, **self._kwargs)
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
backend-1         |     work_item.run()
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 61, in run
backend-1         |     self.future.set_exception(exc)
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 565, in set_exception
backend-1         |     self._invoke_callbacks()
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
backend-1         |     callback(self)
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/events/stream.py", line 329, in _handle_callback_error
backend-1         |     logger.error(
backend-1         |   File "/usr/local/lib/python3.12/logging/__init__.py", line 1568, in error
backend-1         |     self._log(ERROR, msg, args, **kwargs)
backend-1         |   File "/usr/local/lib/python3.12/logging/__init__.py", line 1684, in _log
backend-1         |     self.handle(record)
localstack-1      | 2025-04-01T16:12:30.374  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | 
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/6.json HTTP/1.1" 200 1734
backend-1         | AWS SDK invocation: s3 GetObject
backend-1         | exception calling callback for <Future at 0xffff499726f0 state=finished raised AttributeError>
backend-1         | Traceback (most recent call last):
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
backend-1         |     callback(self)
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/events/stream.py", line 333, in _handle_callback_error
backend-1         |     raise e
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/events/stream.py", line 327, in _handle_callback_error
backend-1         |     fut.result()
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
backend-1         |     return self.__get_result()
backend-1         |            ^^^^^^^^^^^^^^^^^^^
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
backend-1         |     raise self._exception
backend-1         |   File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
backend-1         |     result = self.fn(*self.args, **self.kwargs)
backend-1         |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/threading/__init__.py", line 143, in wrapped_func
backend-1         |     return original_func(*func_args, **func_kwargs)
backend-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/controller/agent_controller.py", line 348, in on_event
backend-1         |     asyncio.get_event_loop().run_until_complete(self._on_event(event))
backend-1         |   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
backend-1         |     return future.result()
backend-1         |            ^^^^^^^^^^^^^^^
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/controller/agent_controller.py", line 362, in _on_event
backend-1         |     await self._handle_action(event)
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/controller/agent_controller.py", line 374, in _handle_action
backend-1         |     await self._handle_message_action(action)
backend-1         |   File "/code/.venv/lib/python3.12/site-packages/openhands/controller/agent_controller.py", line 447, in _handle_message_action
backend-1         |     if self._initial_max_iterations is not None and not self.headless_mode:
backend-1         |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1         | AttributeError: 'AgentController' object has no attribute '_initial_max_iterations'
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/7.json HTTP/1.1" 200 7999
backend-1         | AWS SDK invocation: s3 GetObject
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/8.json HTTP/1.1" 200 1536
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.380  INFO --- [et.reactor-5] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/9.json HTTP/1.1" 200 7451
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.385  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/10.json HTTP/1.1" 200 1846
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.388  INFO --- [et.reactor-4] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/11.json HTTP/1.1" 200 2067
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.392  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/12.json HTTP/1.1" 200 1656
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.397  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/13.json HTTP/1.1" 200 2426
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.400  INFO --- [et.reactor-5] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/14.json HTTP/1.1" 200 1677
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.403  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/15.json HTTP/1.1" 200 1741
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.406  INFO --- [et.reactor-4] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/16.json HTTP/1.1" 200 2379
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.409  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/17.json HTTP/1.1" 200 2136
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.412  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/18.json HTTP/1.1" 200 1800
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.414  INFO --- [et.reactor-5] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/19.json HTTP/1.1" 200 2347
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.418  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/20.json HTTP/1.1" 200 1880
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.421  INFO --- [et.reactor-4] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/21.json HTTP/1.1" 200 1716
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.423  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/22.json HTTP/1.1" 200 1750
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.426  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/23.json HTTP/1.1" 200 1701
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.437  INFO --- [et.reactor-5] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/24.json HTTP/1.1" 200 1782
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.456  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/25.json HTTP/1.1" 200 1712
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.475  INFO --- [et.reactor-4] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/26.json HTTP/1.1" 200 1701
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.482  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/27.json HTTP/1.1" 200 3423
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.489  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/28.json HTTP/1.1" 200 1734
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.494  INFO --- [et.reactor-5] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/29.json HTTP/1.1" 200 1663
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.497  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/30.json HTTP/1.1" 200 1593
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.499  INFO --- [et.reactor-4] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/31.json HTTP/1.1" 200 3110
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.503  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/32.json HTTP/1.1" 200 2280
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.506  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/33.json HTTP/1.1" 200 2017
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.508  INFO --- [et.reactor-5] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/34.json HTTP/1.1" 200 2871
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.510  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/35.json HTTP/1.1" 200 2197
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.513  INFO --- [et.reactor-4] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/36.json HTTP/1.1" 200 3158
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.515  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/37.json HTTP/1.1" 200 185
backend-1         | AWS SDK invocation: s3 GetObject
localstack-1      | 2025-04-01T16:12:30.516  INFO --- [et.reactor-3] localstack.request.aws     : AWS s3.GetObject => 200
backend-1         | http://localstack:4566 "GET /my-log-bucket/sessions/my_session_id/events/38.json HTTP/1.1" 200 2058
@zachmostowsky zachmostowsky added the bug Something isn't working label Apr 1, 2025
@enyst
Copy link
Collaborator

enyst commented Apr 2, 2025

Cc: @chuckbutkus What do you think?

@zachmostowsky
Copy link
Author

I was doing some further testing today and saw this occurring for the LocalFileStore class as well. I have updated the issue to reflect that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@enyst @zachmostowsky