feat(api): Add image multimodal support for LLMNode #17372
Draft
+746
−80
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: -LAN- <[email protected]> # Conflicts: # api/core/model_runtime/entities/message_entities.py
…supported types Signed-off-by: -LAN- <[email protected]>
Enhance `LLMNode` with multimodal capability, introducing support for image outputs. This implementation extracts base64-encoded images from LLM responses, saves them to the storage service, and records the file metadata in the `ToolFile` table. In conversations, these images are rendered as markdown-based inline images. Additionally, the images are included in the LLMNode's output as file variables, enabling subsequent nodes in the workflow to utilize them. To integrate file outputs into workflows, adjustments to the frontend code are necessary. For multimodal output functionality, updates to related model configurations are required. Currently, this capability has been applied exclusively to Google's Gemini models.
Add a detailed notice to guide contributors on avoiding direct usage of the global variable `models.engine.db`. Instead, they are encouraged to apply dependency injection to improve code readability, testability, and maintainability.
Clarify that `ToolFile` now stores not only metadata for files generated by agents but also metadata for files produced by various nodes in a workflow. For instance, it includes metadata for multimodal output files generated by an `LLMNode`.
QuantumGhost
force-pushed
the
feat/support-image-generate-for-gemini
branch
2 times, most recently
from
f78a7db to
b8672a6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Enhance
LLMNodewith multimodal capability, introducing support forimage outputs.
This implementation extracts base64-encoded images from LLM responses,
saves them to the storage service, and records the file metadata in the
ToolFiletable. In conversations, these images are rendered asmarkdown-based inline images.
Additionally, the images are included in the LLMNode's output as
file variables, enabling subsequent nodes in the workflow to utilize them.
To integrate file outputs into workflows, adjustments to the frontend code
are necessary.
For multimodal output functionality, updates to related model configurations
are required. Currently, this capability has been applied exclusively to
Google's Gemini models.
Close #15814.
Screenshots
The image is showed twice. I don't know why. (maybe some issues in frontend code?)
To utilize multimodal output capability, updating to Gemini models is required. Related PR will be submitted later.
Checklist
dev/reformat(backend) andcd web && npx lint-staged(frontend) to appease the lint gods