New resolver mangles development versions #9446

di · 2021-01-13T02:12:39Z

What did you want to do?
Install two dev releases from VCS:

root@6be2bac80999:/# python -m pip install -U pip
Requirement already satisfied: pip in /usr/local/lib/python3.9/site-packages (20.3.1)
Collecting pip
  Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 11.7 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.1
    Uninstalling pip-20.3.1:
      Successfully uninstalled pip-20.3.1
Successfully installed pip-20.3.3

root@6be2bac80999:/# python -m pip install git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=SQLAlchemy git+https://github.com/sqlalchemy/alembic.git@master#egg=alembic
Collecting SQLAlchemy
  Cloning https://github.com/sqlalchemy/sqlalchemy.git (to revision master) to /tmp/pip-install-ydvqru3j/sqlalchemy_a0ec01a8b2ef4a55a02ea0da94e40860
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting alembic
  Cloning https://github.com/sqlalchemy/alembic.git (to revision master) to /tmp/pip-install-ydvqru3j/alembic_846e4d854fec4736b90a50c654d07555
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Cannot install alembic==1.5.0.dev0dev and sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=SQLAlchemy) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=SQLAlchemy)
    alembic 1.5.0.dev0dev depends on SQLAlchemy>=1.3.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Note that pip thinks the version ends in dev0dev instead of dev0, which causes resolution to fail due to the version becoming a LegacyVersion.

When installed separately, pip has no issues:

root@6be2bac80999:/# python -m pip install git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=SQLAlchemy
Collecting SQLAlchemy
  Cloning https://github.com/sqlalchemy/sqlalchemy.git (to revision master) to /tmp/pip-install-flgmx_52/sqlalchemy_43e82260e75e4337957f394c88707f00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: SQLAlchemy
  Building wheel for SQLAlchemy (PEP 517) ... done
  Created wheel for SQLAlchemy: filename=SQLAlchemy-1.4.0b2.dev0-cp39-cp39-linux_x86_64.whl size=1477973 sha256=08f0b2e0c2b3758f85cc15bfee17c951ce3841b5a7375925ecaae7e15b28037a
  Stored in directory: /tmp/pip-ephem-wheel-cache-rtruk_2k/wheels/74/f8/a3/5cdbf2d34871d0a2545a526befcf91053137b290f7bcda75c4
Successfully built SQLAlchemy
Installing collected packages: SQLAlchemy
Successfully installed SQLAlchemy-1.4.0b2.dev0

root@6be2bac80999:/# python -m pip install git+https://github.com/sqlalchemy/alembic.git@master#egg=alembic
Collecting alembic
  Cloning https://github.com/sqlalchemy/alembic.git (to revision master) to /tmp/pip-install-2jze7ib_/alembic_d1eb8411461549fe8ecad1a762ddaf82
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: SQLAlchemy>=1.3.0 in /usr/local/lib/python3.9/site-packages (from alembic) (1.4.0b2.dev0)
Requirement already satisfied: python-editor>=0.3 in /usr/local/lib/python3.9/site-packages (from alembic) (1.0.4)
Requirement already satisfied: Mako in /usr/local/lib/python3.9/site-packages (from alembic) (1.1.3)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.9/site-packages (from alembic) (2.8.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.9/site-packages (from Mako->alembic) (1.1.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/site-packages (from python-dateutil->alembic) (1.15.0)
Building wheels for collected packages: alembic
  Building wheel for alembic (PEP 517) ... done
  Created wheel for alembic: filename=alembic-1.5.0.dev0-py2.py3-none-any.whl size=146794 sha256=27f7c98291150cdc568d3fb6c065cba1ff5f97daa509bca61876c7eeeb8b2c76
  Stored in directory: /tmp/pip-ephem-wheel-cache-upxtfqqg/wheels/5d/f2/cc/a7c83f7926ce4195064cc30f8195c6d1c53d9f6ee0100c0bd6
Successfully built alembic
Installing collected packages: alembic
Successfully installed alembic-1.5.0.dev0

Installing both together works with the old resolver:

root@6be2bac80999:/# python -m pip install pip==20.2.4
Collecting pip==20.2.4
  Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 10.0 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.3
    Uninstalling pip-20.3.3:
      Successfully uninstalled pip-20.3.3
Successfully installed pip-20.2.4

root@6be2bac80999:/# python -m pip install git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=SQLAlchemy git+https://github.com/sqlalchemy/alembic.git@master#egg=alembic
Collecting SQLAlchemy
  Cloning https://github.com/sqlalchemy/sqlalchemy.git (to revision master) to /tmp/pip-install-h4h2iix6/sqlalchemy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting alembic
  Cloning https://github.com/sqlalchemy/alembic.git (to revision master) to /tmp/pip-install-h4h2iix6/alembic
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting Mako
  Downloading Mako-1.1.3-py2.py3-none-any.whl (75 kB)
     |████████████████████████████████| 75 kB 4.6 MB/s
Collecting python-editor>=0.3
  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Collecting python-dateutil
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 12.9 MB/s
Collecting MarkupSafe>=0.9.2
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Collecting six>=1.5
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: SQLAlchemy, alembic, MarkupSafe
  Building wheel for SQLAlchemy (PEP 517) ... done
  Created wheel for SQLAlchemy: filename=SQLAlchemy-1.4.0b2.dev0-cp39-cp39-linux_x86_64.whl size=1477841 sha256=72645d87d755079da654209fcd060650dba05e7b720a2a924afa9b956fdfe9f9
  Stored in directory: /tmp/pip-ephem-wheel-cache-s53kkwde/wheels/74/f8/a3/5cdbf2d34871d0a2545a526befcf91053137b290f7bcda75c4
  Building wheel for alembic (PEP 517) ... done
  Created wheel for alembic: filename=alembic-1.5.0.dev0-py2.py3-none-any.whl size=146794 sha256=8c2f69bb60171e590a4bff92624fc651eadd0e4e916c985e73cda14277a5239d
  Stored in directory: /tmp/pip-ephem-wheel-cache-s53kkwde/wheels/5d/f2/cc/a7c83f7926ce4195064cc30f8195c6d1c53d9f6ee0100c0bd6
  Building wheel for MarkupSafe (setup.py) ... done
  Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-cp39-cp39-linux_x86_64.whl size=32205 sha256=e0ae4d57d299916a810883ad6fd41a71cf39ad49cd44a760628dca162e6e2866
  Stored in directory: /root/.cache/pip/wheels/e0/19/6f/6ba857621f50dc08e084312746ed3ebc14211ba30037d5e44e
Successfully built SQLAlchemy alembic MarkupSafe
Installing collected packages: SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic
Successfully installed Mako-1.1.3 MarkupSafe-1.1.1 SQLAlchemy-1.4.0b2.dev0 alembic-1.5.0.dev0 python-dateutil-2.8.1 python-editor-1.0.4 six-1.15.0
WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

Additional information

This I bisected this to determine that this issue is present in all commits where the new resolver is available.

Originally reported here: https://twitter.com/zzzeek/status/1349138336242413568

The text was updated successfully, but these errors were encountered:

uranusjr · 2021-01-13T03:44:33Z

This seems to be a setuptools bug. I can reproduce the version mangling with the following:

$ git clone https://github.com/sqlalchemy/alembic.git
$ cd alembic
$ py -c 'import os; from setuptools.build_meta import *; prepare_metadata_for_build_wheel(os.getcwd())'
$ rg '^Version:' < alembic.egg-info/PKG-INFO
Version: 1.5.0.dev0dev

BUT this only happens with prepare_metadata_for_build_wheel; build_wheel creates a wheel with the correct version. When each package is installed separately (without version constraints), any version string is allowed, so the resolver lets 1.5.0.dev0dev pass. The later built wheel is “correctly” installed with version set as 1.5.0.dev0.

uranusjr · 2021-01-13T09:50:33Z

This makes me think, maybe pip should add checks to ensure tools are following PEP 517 rules? PEP 517 explicitly prohibits this kind of issues:

If [metadata_directory] is provided, then build_wheel MUST produce a wheel with identical metadata.

The error is (fortunately) caught during resolution in this case because prepare_metadata_for_build_wheel goes wrong, without build_wheel silently “correcting” the error. Debugging would be far less straightforward if the error happens the other way around, i.e. prepare_metadata_for_build_wheel works correctly, but a subsequent build_wheel generates invalid metadata.

zzzeek · 2021-01-13T13:39:28Z

hey -

any workaround for this right now? I will have to lower the pip version I'm using which might be involved since I am running tests against openstack projects that I'm altering from the outside. every change is another sed operation basically.

uranusjr · 2021-01-13T14:42:59Z

One workaround would be to build the wheels separately and install from those instead. This avoids the problematic prepare_metadata_for_build_wheel.

# This builds wheels into a subdirectory ./wheels
pip wheel --no-deps -w ./wheels \
	git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=SQLAlchemy \
	git+https://github.com/sqlalchemy/alembic.git@master#egg=alembic

# Install the built wheels.
pip install ./wheels/*.whl

zzzeek · 2021-01-13T15:29:40Z

ultimately this has to all run from a tox.ini file and I think years ago I might have been doing it more that way, actually forcing pip to only install from a local cache (which meant I had to reinvent the whole thing for hundreds of openstack dependencies) then when I could put git URLs in requirements files I was able to simplify things...so...im going to see if I can force the pip version down first.

Waiting for pypa/pip#9446 to be resolved we will pin pip to 19.3.1 for now this still might not work because it might need newer virtualenv and tox installations on containers to work. it's very hard to follow which tool does what under what conditions.

di · 2021-01-15T20:54:12Z

any workaround for this right now?

For a quick fix you can change tag_build = dev to tag_build = dev0.

I submitted a PR to properly fix this here: pypa/setuptools#2533

CaselIT · 2021-04-29T07:39:41Z

any workaround for this right now?

For a quick fix you can change tag_build = dev to tag_build = dev0.

I submitted a PR to properly fix this here: pypa/setuptools#2533

Sadly that pr was rolled back. This is still an issue in the 21.1.x version of pip

uranusjr · 2021-04-29T08:30:59Z

pypa/setuptools#2529 contains more information.

I'm going to close this issue since the issue is in setuptools, and there is nothing pip can do here.

uranusjr mentioned this issue Jan 13, 2021

tag_build breaks prepare_metadata_for_build_wheel pypa/setuptools#2529

Open

uranusjr added the C: PEP 517 impact Affected by PEP 517 processing label Jan 13, 2021

uranusjr mentioned this issue Jan 30, 2021

build: implement metadata preparation hook pypa/build#217

Merged

uranusjr closed this as completed Apr 29, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New resolver mangles development versions #9446

New resolver mangles development versions #9446

di commented Jan 13, 2021

uranusjr commented Jan 13, 2021 •

edited

Loading

uranusjr commented Jan 13, 2021

zzzeek commented Jan 13, 2021

uranusjr commented Jan 13, 2021 •

edited

Loading

zzzeek commented Jan 13, 2021

di commented Jan 15, 2021

CaselIT commented Apr 29, 2021

uranusjr commented Apr 29, 2021

New resolver mangles development versions #9446

New resolver mangles development versions #9446

Comments

di commented Jan 13, 2021

uranusjr commented Jan 13, 2021 • edited Loading

uranusjr commented Jan 13, 2021

zzzeek commented Jan 13, 2021

uranusjr commented Jan 13, 2021 • edited Loading

zzzeek commented Jan 13, 2021

di commented Jan 15, 2021

CaselIT commented Apr 29, 2021

uranusjr commented Apr 29, 2021

uranusjr commented Jan 13, 2021 •

edited

Loading

uranusjr commented Jan 13, 2021 •

edited

Loading