Impossible resolution for semingly resolvable requirements

[fmigneault]

Description

My requirements specify the following dependencies:

billiard!=2.*; sys_platform != "win32" 
celery[mongodb]>=5.1; sys_platform != "win32" and python_version >= "3.7"

Pip eventually reports the following and fails installation:

ERROR: Cannot install billiard!=2.*, celery[mongodb]==5.1.0, celery[mongodb]==5.1.1, celery[mongodb]==5.1.2, celery[mongodb]==5.2.0, celery[mongodb]==5.2.1, celery[mongodb]==5.2.2 and celery[mongodb]==5.2.3 because these package versions have conflicting dependencies.

The conflict is caused by:
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    The user requested billiard!=2.*
    celery[mongodb] 5.2.3 depends on billiard<4.0 and >=3.6.4.0
    The user requested billiard!=2.*
    celery[mongodb] 5.2.2 depends on billiard<4.0 and >=3.6.4.0
    The user requested billiard!=2.*
    celery[mongodb] 5.2.1 depends on billiard<4.0 and >=3.6.4.0
    The user requested billiard!=2.*
    celery[mongodb] 5.2.0 depends on billiard<4.0 and >=3.6.4.0
    The user requested billiard!=2.*
    celery[mongodb] 5.1.2 depends on billiard<4.0 and >=3.6.4.0
    The user requested billiard!=2.*
    celery[mongodb] 5.1.1 depends on billiard<4.0 and >=3.6.4.0
    The user requested billiard!=2.*
    celery[mongodb] 5.1.0 depends on billiard<4.0 and >=3.6.4.0

I'm not sure if this is related to #10873 since an extra is also involved in my case.

If I remove billiard entirely to let celery dependencies resolve it by itself, the installation instead times out due to too many combinations being explored.
I have also tried with other variants, such as billiard>=3, billiard<4.0,>=3.6.4.0 (exactly what celery asks for) and so on.
No alternative seem to work. Pip always consider them as conflicting.

Expected behavior

There shouldn't be any conflict / impossible resolution for valid version ranges.

pip version

22.0.4

Python version

3.6, 3.7, possibly following 3.x as well

OS

Ubuntu 20.04

How to Reproduce

Pip install requirements.txt file with the following:

billiard!=2.*; sys_platform != "win32" 
celery[mongodb]>=5.1; sys_platform != "win32" and python_version >= "3.7"

Full requirements file I'm actually using: https://github.com/crim-ca/weaver/blob/e56990e3b34814ae2b4882045eda1b5374b64584/requirements.txt

Output

pip.log

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[samir321-pixel]

Facing same issue :-(

[notatallshaw]

Facing same issue :-(

Exactly the same as listed above with the same requirements? Or do you have different requirements and therefore could provide different steps to reproduce?

I believe this issue is/was a reporting issue, the provided requirements are impossible to resolve but pop doesn't clearly report why.

[sparrowt]

I posted a similar issue here: (ResolutionImpossible despite no apparent conflict in version, in fact no version was specified at all)
https://stackoverflow.com/questions/74190982/resolutionimpossible-from-pip-despite-no-apparent-version-conflict/

In my case the problem was that no wheel was available for the required platform (e.g. windows/linux) - for me this was due to using --no-index --find-links LOCALDIR and not putting the correct wheel in there, but I could imagine the same might occur if using PyPI and the package author has not published a wheel for the platform in question.

Not sure if that is the same cause as for the OP, but either way pip is not giving a correct (or helpful) message here, because there aren't any "conflicting dependencies", just a missing wheel.

I guess this is similar conceptually to #6526 although a different code path (given substantially different output).

[fmigneault]

I have managed to install both libraries by themselves with versions that fulfil the requirements.
The original error only happened when the full set of requirements was specified.
Therefore, the issue does not seem to be related to missing wheels.
A clearer message would indeed help to track down the cause of resolution conflicts, since it does not seem to be due to billiard and celery themselves, but maybe another underlying package dependency that is not listed in the printed error.

❯ pip show billiard
Name: billiard
Version: 3.6.4.0
Summary: Python multiprocessing fork with improvements and bugfixes
Home-page: https://github.com/celery/billiard
Author: R Oudkerk / Python Software Foundation
Author-email: [email protected]
License: BSD
Location: /home/francis/dev/miniconda/envs/weaver-py37/lib/python3.7/site-packages
Requires: 
Required-by: celery, weaver
❯ pip show celery
Name: celery
Version: 5.2.7
Summary: Distributed Task Queue.
Home-page: http://celeryproject.org
Author: Ask Solem
Author-email: [email protected]
License: BSD
Location: /home/francis/dev/miniconda/envs/weaver-py37/lib/python3.7/site-packages
Requires: billiard, click, click-didyoumean, click-plugins, click-repl, kombu, pytz, vine
Required-by: pyramid-celery, weaver

[notatallshaw]

The original error only happened when the full set of requirements was specified.
Therefore, the issue does not seem to be related to missing wheels.
A clearer message would indeed help to track down the cause of resolution conflicts, since it does not seem to be due to

Seems there is some improvement in the original error message since you originally posted, either because of improvements in Pip or improvements in the real world dependencies. I just attempted your original requirement now on the latest version of Pip, on Ubuntu, using Python 3.7. The full Error message I got was:

The conflict is caused by:
cwltool 2.0.20200107113851 depends on rdflib<4.3.0 and >=4.2.2
prov 1.5.1 depends on rdflib>=4.2.1
rdflib-jsonld 0.6.2 depends on rdflib>=5.0.0
cwltool 2.0.20200107113851 depends on rdflib<4.3.0 and >=4.2.2
prov 1.5.1 depends on rdflib>=4.2.1
rdflib-jsonld 0.6.1 depends on rdflib>=5.0.0

The issue appears to be cwltool depends on rdflib<4.3.0, and rdflib-jsonld depends on rdflib>=5.0.0.

This seems reasonable?

[fmigneault]

@notatallshaw
Yes. cwltool had too strict requirements that have been fixed in more recent 3.x versions.
I did not see the problem related to celery/billiard anymore in more recent attempts.

[notatallshaw]

This specific issue was resolved. Please do open issues if you find problems with dependency resolution, there are known issues being worked on, and more examples are always helpful.