Examples of ResolutionTooDeep in Pip
#13281
Labels
C: dependency resolution
About choosing which dependencies to install
C: error messages
Improving error messages
Comments
notatallshaw
added
C: dependency resolution
1 task
Only Python 3.12 fails Other images built properly. Example failure: https://github.com/apache/airflow/actions/runs/14692610732/job/41231149374#step:9:130 Attempting to see if apache/airflow#49853 would help:
Command:
The most interesting part of the output is: |
|
Thanks, I'll take a look as soon as I can, probably later today. |
|
I see you have applied a workaround, which is good because I think it might take me a little time and understanding your build system to produce an MRE, as testing with the latest releases packages it doesn't appear to hit this: |
|
We actually have another iteration on that. Yesterday we started to hit another one (also only with Python 3.12): https://github.com/apache/airflow/actions/runs/14698736309/job/41248421057 I attempt (wild guessing) to limit it with: BTW. The way to reproduce it is fairly easy when you know how. Here is a detailed set of instructions:
uv tool install -e ./dev/breeze # pipx should also work
breeze release-management prepare-provider-distributions --distributions-list-file ./prod_image_installed_providers.txt --distribution-format wheel --include-not-ready-providers --skip-tag-check
breeze release-management prepare-airflow-distributions --distribution-format wheel
breeze release-management prepare-task-sdk-distributions --distribution-format wheel
breeze release-management prepare-airflow-ctl-distributions --distribution-format wheelThat will prepare all packages in
mv dist/*.whl docker-context-files/
breeze prod-image build --install-distributions-from-context --airflow-constraints-mode constraints-source-providers --verboseWith
The interesting thing is that we know there is a possible resolution for those packages. They are all part of the same workspace in airflow repo: uv sync --all-packages --python 3.12Will sync even more than that - it will resolve ALL the provider packages + ALL even their development dependencies - and what we try to install in the PROD image is merely a subset of all the providers. I hope it helps :) |
|
BTW. I think it might be involved with some protobuf incompatibilities. The apache/airflow#49860 is not going to work - because of protobuf conflicts - I am looking at it |
The same requirements resolve with pip 25.0 (in around 13 minutes on the same hardware). I've cut out a bunch of internal specifics and constraints, but this is still a pretty "organic" case and not a minimum reproduction. Please let me know if other information would be helpful. |
|
Thanks @cburroughs, that's plenty of info, I'll start investigating, but I'm not likely to give you an exact workaround for a few days, and any pip optimization could take some time. For the moment I can only recommend you either use |
Thanks for investigating! Per the 'not for help' disclaimer at the top, my hope is that a real world example helps improve pip and I'm not expecting one uncanny trick that unblocks just me. I'm sure there are promising permutations of constraints around boto and other 'usual suspects' I can try before upgrading to a brand new pip becomes pressing. |
After more fixes - still "resolution too deep" - but with slightly different trace - willl continue limiting stuff
https://github.com/apache/airflow/actions/runs/14721786685/job/41318681642 Update: attempted to limit portalocker here: apache/airflow#49936 |
|
And yeah @notatallshaw -> we do understand it's not an easy issue to solve, and airflow and the way we have 700+ deps is a bit of an outlier, but we'll keep trying to make our deps a bit more Hopefully together we can improve resolver :) |
|
Yeah, sorry for this hit you multiple times, I assume all of these are caused by me limiting how much resolvelib will backjump. Unfortunately it's approach was unsound. Once I've collected all of these in to my test scenario I'll play around with various changes to heuristics to see if they make an improvements. A true fix might not come until pip adopts a CDCL or CDNL algorithm like pubgrub, like uv and poetry use. Which is my long term goal, but requires some information that pip and packaging aren't providing yet, I don't have any sort of timeline. |
Happy to help with more "playing" with it - it's not a huge problem for us - though we would love to get it sorted out soon-ish |
|
Hmm - even with latest attemp we still have "too deep". https://github.com/apache/airflow/actions/runs/14732688845/job/41355638430 This time without a clear "contender: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🐍 Collecting Examples of
ResolutionTooDeepin PipThis post is not for help, but for collecting examples of where Pip's resolver fails with
ResolutionTooDeep. Your contributions will help test future improvements to the resolver's behavior.✅ What to Include
When submitting an example, please provide the following:
pip --version)python --version)pip freezepyproject.toml,requirements.txt, or other relevant files (if applicable)pyproject.toml, please include thedependenciesvalue from the[project]section.📝 Example Submission
📌 Why This Matters
By collecting real-world examples, we can better understand when and why the resolver goes too deep and improve Pip's behavior.
Thank you for helping us make Pip better!
The text was updated successfully, but these errors were encountered: