Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix retrieve input restriction. #16744

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

gt-yuanmeijie
Copy link

When calling the interface, only check for the existence of the query without considering character limitations.

Closes #16724

Summary

Resolves #<16724>

Screenshots

| Before | After |

Checklist

Important

Please review the checklist below before submitting your pull request.

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

When calling the interface, only check for the existence of the query without considering character limitations.

Closes langgenius#16724
@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. 🐞 bug Something isn't working labels Mar 25, 2025
@@ -42,6 +42,11 @@ def get_and_validate_dataset(dataset_id: str):
def hit_testing_args_check(args):
HitTestingService.hit_testing_args_check(args)

@staticmethod
def retrieval_args_check(args):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hit-testing is the same with retrieve.

  • retrieve is the new endpoint
  • We keep the hit-testing mainly because we do not want to break existing applications.

Copy link
Member

@crazywoola crazywoola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May I ask how many characters do you need, it seems 250 is enough for most of the cases.
Are there any examples?

  • Dead code should also be removed
  • Please fix the lint errors as well.

@JohnJyong
Copy link
Collaborator

query's max length limit is required , you can make the value as an env variable, so you can changed it in your local server.
I would like to ask in what scenarios the query needs to be more than 250 characters? @gt-yuanmeijie

@gt-yuanmeijie
Copy link
Author

Sorry for the delayed response. I had some matters to attend to today.
My main development language is Java, and yesterday I submitted the code using IntelliJ IDEA, which caused some formatting issues.
I might need to use these two scenarios:

Use a knowledge base for dynamic semantic deduplication.
Dynamically use different knowledge bases based on different conditions, and then call an agent to respond.
The approach for the first scenario is as follows: based on the user's query, call the retrieve interface. If there are results, return the data with the highest match. If not, call the semantic analysis agent and insert the agent's result as a segment of a dataset into the knowledge base.

The reason for the second scenario is that we need to choose the knowledge base from three dimensions, one of which increases over time. We do not want to frequently modify the application (modifying the application carries risks). My idea is to store these configuration relationships in the database. When a user asks a question, retrieve the desired knowledge base from the database based on different dimensions, and then call the app with the result returned by the retrieve interface as a parameter.

The user's questions cannot be controlled, and currently, the limit of 250 characters is indeed a bit short.

I originally intended to write a separate RetrieveApi to differentiate between retrieve and hit_test, but I am not that familiar with Python web development, so I did not make any changes. Additionally, hit_testing_args_check is used in the api/controllers/datasets/hit_testing.py file. This time, I have only made modifications to the api/controllers/service_api/dataset/hit_testing.py file.

@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Mar 26, 2025
@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
3 participants