update snippets for image-text-to-text models for transformers models

[Vaibhavs10]

No description provided.

[pcuenca]

hmm, I think this line is still out of the conversational path. How about something like this? (forgive mistakes, typing in this text box)

		const pipelineSnippet = ["# Use a pipeline as a high-level helper", "from transformers import pipeline", ""];
		pipelineSnippet.push(`pipe = pipeline("${model.pipeline_tag}", model="${model.id}"` + remote_code_snippet + ")");

		if (model.config?.tokenizer_config?.chat_template && model.tags.includes("conversational")) {
			if (model.tags.includes("image-text-to-text")) {
				pipelineSnippet.push(
					"messages = [",
					[
						"    {",
						'        "role": "user",',
						'        "content": [',
						'            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},',
						'            {"type": "text", "text": "What animal is on the candy?"}',
						"        ]",
						"    },",
					].join("\n"),
					"]"
				);
			} else {
				pipelineSnippet.push("messages = [", '    {"role": "user", "content": "Who are you?"},', "]");
			}
			pipelineSnippet.push("pipe(messages)");
		}

		return [pipelineSnippet.join("\n"), autoSnippet];

[Vaibhavs10]

It isn't no? https://gist.github.com/Vaibhavs10/b525c1ae4982c04fe0202c2f78ac2f03

the pipeline instantiation and the pipe(messages) is the same for both

(my JS sucks so could very well be a skill issue)

[pcuenca]

lol, I'm probably blind by now, but this is the way I'm seeing it:

i.e., if there is no chat_template, there's no messages.

Also, if conversational is not present (a mistake, probably) we wouldn't have messages either, that's why I suggested to require it in the outer if too.

[Wauplin]

I suggested another variation based on your previous comments here: https://gist.github.com/Vaibhavs10/b525c1ae4982c04fe0202c2f78ac2f03?permalink_comment_id=5569573#gistcomment-5569573

Changes are:

• the pipe = pipeline(... line should be added at the beginning => no matter the pipeline_tag we want to instantiate the pipeline
• the pipe(messages) line should be added only if conversational mode as @pcuenca mentioned above
• I would not check model.config?.tokenizer_config?.chat_template manually but rely only on the model.tags.includes("conversational") condition. This makes the logic simpler and it should be equivalent since moon-landing is automatically adding conversational when chat template is detected (see private link) (it's even better because moon-landing checks for both tokenizer_config and processor_config and is maintained to be future-proof)

[julien-c]

yes^

[Vaibhavs10]

I'm stupid for not looking at that code block - sorry for being so dense!

[Wauplin]

Looks good!