Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update conversation metadata with cost and token metrics #7561

Merged
merged 5 commits into from
Apr 7, 2025

Conversation

rbren
Copy link
Collaborator

@rbren rbren commented Mar 27, 2025

This PR updates the conversation metadata to include cost/token metrics from events.

Changes

  1. Added new fields to ConversationMetadata class:

    • accumulated_cost: Tracks the total cost of the conversation
    • prompt_tokens: Tracks the number of prompt tokens used
    • completion_tokens: Tracks the number of completion tokens used
    • total_tokens: Tracks the total number of tokens used
  2. Renamed _update_timestamp_for_conversation to _update_conversation_for_event to better reflect its expanded purpose

  3. Modified _create_conversation_update_callback to pass the event to the callback function

  4. Updated the function to extract and store metrics from the event when available

All tests are passing with these changes.


To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:cd25e32-nikolaik   --name openhands-app-cd25e32   docker.all-hands.dev/all-hands-ai/openhands:cd25e32

if hasattr(metrics, 'accumulated_token_usage'):
token_usage = metrics.accumulated_token_usage
conversation.prompt_tokens += token_usage.prompt_tokens
conversation.completion_tokens += token_usage.completion_tokens
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like _accumulated_token_usage is already a TokenUsage object which includes the additions:
https://github.com/All-Hands-AI/OpenHands/pull/7511/files#diff-9010f8fab42fcc1b1464f428310e3ef8f93d07c22b6b481bedbbc5a0d864386aR132

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch! reverting

@rbren rbren marked this pull request as ready for review April 7, 2025 16:34
@rbren rbren enabled auto-merge (squash) April 7, 2025 16:43
@rbren rbren merged commit 1549927 into main Apr 7, 2025
14 checks passed
@rbren rbren deleted the update-conversation-metrics branch April 7, 2025 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants