[Usage]: How can I quickly obtain the number of prompt tokens containing multimodal data?

[yansh97]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

The /tokenize API can only return the number of prompt tokens that contain text and multimodal placeholders, but cannot return the actual number of prompt tokens. @DarkLight1337

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

It's not possible yet. Help is welcome!

[DarkLight1337]

We need to call the processor from the API server in order to get the multimodal tokens.

[chaunceyjiang]

I can try fix this issue.

[w013nad]

+1 Would love to have this feature

[chaunceyjiang]

import requests

text = "hello world!"
model_name = "Qwen/Qwen2.5-VL-7B-Instruct"
# model_name = ""
url = "http://localhost:8000/tokenize"
data = {"model": model_name, "prompt": text}
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

messages = [{
    "role":
    "user",
    "content": [
        {
            "type": "text",
            "text": "What's in this image?"
        },
        {
            "type": "image_url",
            "image_url": {
                "url": image_url
            },
        },
    ],
}]

data = {"model": model_name, "messages": messages}

response = requests.post(url, json=data)
result = response.json()

print("Token IDs:", result["tokens"])
print("Token count:", result["count"])

output:

Token IDs: [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 3838, 594, 304, 419, 2168, 30, 151652, 151655, 151653, 151645, 198, 151644, 77091, 198]
Token count: 28

I have a test case, as shown above. I understand that the generated token IDs contain placeholders like <image>, but do not include the actual image content. This issue requires the actual token IDs, i.e., including the image content itself, not just the placeholder.

Hi, @DarkLight1337 I need some help. I'm not sure if I understand this correctly, but it seems that the patch embedding of the image itself doesn't have token IDs — the embedding is processed inside the model.

How should I handle this?

[DarkLight1337]

Yes, that's why I said you need to apply the multi-modal processor instead of just the tokenizer.