Skip to content

[CI][Bugfix] Fix bad tolerance for test_batch_base64_embedding #16221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 9, 2025

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented Apr 8, 2025

Previously the test was doing equality on float values, which is always a bad idea. See an example of flaky test https://buildkite.com/vllm/fastcheck/builds/19776/steps?jid=0196126c-0cdf-4b29-9363-d925adaf681a#0196126c-0cdf-4b29-9363-d925adaf681a/6-13621

This PR changes it to use allclose

Copy link

github-actions bot commented Apr 8, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Copy link
Member

@ywang96 ywang96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix 🙏 !

@mgoin mgoin added bug Something isn't working ci/build labels Apr 8, 2025
@mgoin mgoin changed the title Fix bad tolerance for test_batch_base64_embedding [CI][Bugfix] Fix bad tolerance for test_batch_base64_embedding Apr 8, 2025
@ywang96 ywang96 added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 8, 2025
@DarkLight1337 DarkLight1337 enabled auto-merge (squash) April 8, 2025 01:54
@DarkLight1337
Copy link
Member

DarkLight1337 commented Apr 8, 2025

Seems that the default tolerance isn't quite enough. Does this issue appear when we use V0?

Copy link
Member

@hmellor hmellor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mgoin
Copy link
Member Author

mgoin commented Apr 8, 2025

Does this issue appear when we use V0?

I think embedding doesn't work on V1 yet, so this should be V0

Copy link
Contributor

@maxdebayser maxdebayser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I faced the same problem in my PR #16188

I used the check_embeddings_close function from models.embedding.utils. That function uses cosine similarity, but I think allclose should be fine as well.

@DarkLight1337 DarkLight1337 merged commit 87b4ac5 into vllm-project:main Apr 9, 2025
24 checks passed
nishith-fujitsu pushed a commit to nishith-fujitsu/vllm that referenced this pull request Apr 9, 2025
nishith-fujitsu pushed a commit to nishith-fujitsu/vllm that referenced this pull request Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working ci/build ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants