Skip to content

[Feature Request] Support Gemini’s New Multimodal Output #15814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
5 tasks done
laipz8200 opened this issue Mar 14, 2025 · 3 comments · May be fixed by #16085 or #17372
Open
5 tasks done

[Feature Request] Support Gemini’s New Multimodal Output #15814

laipz8200 opened this issue Mar 14, 2025 · 3 comments · May be fixed by #16085 or #17372
Assignees
Labels
💪 enhancement New feature or request 🌊 feat:workflow Workflow related stuff.

Comments

@laipz8200
Copy link
Member

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Background

Google recently released the Gemini 2.0 Flash Experimental API, which includes support for generating images. This is an important step toward broader multimodal capabilities, and we anticipate future support for audio, video, and documents as well.

To keep Dify aligned with these advancements, we propose updating the backend to support Gemini’s new output formats while rendering them in Markdown format on the frontend.

Feature Scope

  • Backend Update Only: No UI/UX changes are required; the frontend will render responses using Markdown.
  • Support for Image Generation (Initial Focus): Since Gemini 2.0 Flash Exp already supports image generation, we should prioritize integrating this feature first.
  • Extendable for Future Media Outputs: The implementation should be flexible enough to accommodate audio, video, and documents once Gemini enables them.

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
@laipz8200 laipz8200 self-assigned this Mar 14, 2025
@dosubot dosubot bot added the 💪 enhancement New feature or request label Mar 14, 2025
@laipz8200 laipz8200 linked a pull request Mar 18, 2025 that will close this issue
5 tasks
@QuantumGhost QuantumGhost added the 🌊 feat:workflow Workflow related stuff. label Mar 20, 2025 — with Linear
@zhangever
Copy link
Contributor

I need this feature, too. How's the cr going?

@laipz8200
Copy link
Member Author

@zhangever Thanks for your comment! We're currently testing this feature, and it shouldn't be too long before it's ready for you. 😌

@JacoBezuidenhout
Copy link

Heyy everyone! Thanks sooo much for the effort you guys put into this! Going to be amazing!

@QuantumGhost QuantumGhost linked a pull request Apr 3, 2025 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💪 enhancement New feature or request 🌊 feat:workflow Workflow related stuff.
Projects
None yet
4 participants