-
Notifications
You must be signed in to change notification settings - Fork 29.2k
[Qwen2.5-VL] Fix empty string input crash in processor #38421
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…to ensure that `Qwen2Tokenizer` correctly handles empty string inputs...
test(Qwen2Tokenizer): Add regression test for empty string input dtype (#38417) This commit introduces a regression test within The test uses the "Qwen/Qwen2-0.5B" model and asserts that the This test serves to prevent future regressions and will pass once |
Hi maintainers, I've pushed an update to this PR. Here's the current status of the CI checks:
This PR, which aims to add this regression test, is exhibiting the expected behavior for its core test (i.e., identifying the existing bug). It should now be ready for review. Thanks! |
Thank you for the PR, but unfortunately it's a duplicate of #36555! It's unfortunate - your code looks good and I hope this doesn't discourage you from contributing in future |
Hi @Rocketknight1, thank you for the quick review and the kind words! I understand that this PR is a duplicate of #36555. I appreciate the feedback and the opportunity to go through the contribution process. It was a great learning experience for me. I'll continue exploring other open issues and look forward to contributing more in the future! Thanks again |
This PR fixes #38417.
When passing an empty string to the Qwen2.5 tokenizer with return_tensors="pt", the original output was a float32 tensor.
This patch ensures a consistent torch.long dtype by returning torch.empty((1, 0), dtype=torch.long) for empty input.
A test is included to validate the fix.
This is my first contribution to 🤗 Transformers. Happy to help and open to feedback!