Pip does not upgrade from Git URL when changing commit but not version

[aphedges]

Environment

• pip version: 20.3
• Python version: 3.7.9
• OS: macOS 10.15.7

What did you want to do?

Upgrade a package to a specific Git commit when both commits have the same package version

I carried out the following steps:

• Create new virtual environment
• pip install --upgrade pip setuptools (update existing packages)
• pip install git+https://github.com/isi-vista/kairos-yaml.git@bbd79bde548fc2ca79c4d053cb41116145a05acc (install old version)
• pip install --upgrade git+https://github.com/isi-vista/kairos-yaml.git@defa596e225848ddeabd26dfb21b9c808dee6240 (upgrade unsuccessfully)
• pip --use-deprecated=legacy-resolver install --upgrade git+https://github.com/isi-vista/kairos-yaml.git@defa596e225848ddeabd26dfb21b9c808dee6240 (upgrade successfully when using the legacy resolver)

Output

$ pip install git+https://github.com/isi-vista/kairos-yaml.git@bbd79bde548fc2ca79c4d053cb41116145a05acc
Collecting git+https://github.com/isi-vista/kairos-yaml.git@bbd79bde548fc2ca79c4d053cb41116145a05acc
  Cloning https://github.com/isi-vista/kairos-yaml.git (to revision bbd79bde548fc2ca79c4d053cb41116145a05acc) to /private/var/folders/n_/ns29p9rx20ng6cn6lcpxj6mr0000gp/T/pip-req-build-if9h1c96
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting lazy-object-proxy>=1.4.3
  Using cached lazy_object_proxy-1.5.2-cp37-cp37m-macosx_10_15_x86_64.whl
Collecting pydantic>=1.7.2
  Using cached pydantic-1.7.3-cp37-cp37m-macosx_10_9_x86_64.whl (2.3 MB)
Collecting PyYAML>=5.3.1
  Using cached PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl
Collecting requests>=2.25.0
  Using cached requests-2.25.0-py2.py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.11.8-py2.py3-none-any.whl (155 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Building wheels for collected packages: kairos-yaml
  Building wheel for kairos-yaml (PEP 517) ... done
  Created wheel for kairos-yaml: filename=kairos_yaml-0.0.1-py3-none-any.whl size=26204 sha256=1b9c95946162ecc941ab730d0bfcd373b969c80b65d88bf76a18a0af516c1b74
  Stored in directory: /Users/ahedges/Library/Caches/pip/wheels/29/7c/ee/dd07864df8f24dbe02b9a20835a35ee8eb69e43dad77f42ea7
Successfully built kairos-yaml
Installing collected packages: urllib3, idna, chardet, certifi, requests, PyYAML, pydantic, lazy-object-proxy, kairos-yaml
Successfully installed PyYAML-5.3.1 certifi-2020.11.8 chardet-3.0.4 idna-2.10 kairos-yaml-0.0.1 lazy-object-proxy-1.5.2 pydantic-1.7.3 requests-2.25.0 urllib3-1.26.2
$ pip install --upgrade git+https://github.com/isi-vista/kairos-yaml.git@defa596e225848ddeabd26dfb21b9c808dee6240
Collecting git+https://github.com/isi-vista/kairos-yaml.git@defa596e225848ddeabd26dfb21b9c808dee6240
  Cloning https://github.com/isi-vista/kairos-yaml.git (to revision defa596e225848ddeabd26dfb21b9c808dee6240) to /private/var/folders/n_/ns29p9rx20ng6cn6lcpxj6mr0000gp/T/pip-req-build-z2i779ve
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: PyYAML>=5.3.1 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (5.3.1)
Requirement already satisfied: requests>=2.25.0 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (2.25.0)
Requirement already satisfied: lazy-object-proxy>=1.4.3 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (1.5.2)
Requirement already satisfied: pydantic>=1.7.2 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (1.7.3)
Requirement already satisfied: certifi>=2017.4.17 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (2020.11.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (1.26.2)
$ pip --use-deprecated=legacy-resolver install --upgrade git+https://github.com/isi-vista/kairos-yaml.git@defa596e225848ddeabd26dfb21b9c808dee6240
Collecting git+https://github.com/isi-vista/kairos-yaml.git@defa596e225848ddeabd26dfb21b9c808dee6240
  Cloning https://github.com/isi-vista/kairos-yaml.git (to revision defa596e225848ddeabd26dfb21b9c808dee6240) to /private/var/folders/n_/ns29p9rx20ng6cn6lcpxj6mr0000gp/T/pip-req-build-toktciz8
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied, skipping upgrade: lazy-object-proxy>=1.4.3 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (1.5.2)
Requirement already satisfied, skipping upgrade: pydantic>=1.7.2 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (1.7.3)
Requirement already satisfied, skipping upgrade: requests>=2.25.0 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (2.25.0)
Requirement already satisfied, skipping upgrade: PyYAML>=5.3.1 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from kairos-yaml==0.0.1) (5.3.1)
Requirement already satisfied, skipping upgrade: urllib3<1.27,>=1.21.1 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (1.26.2)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (2020.11.8)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /Users/ahedges/.pyenv/versions/3.7.9/envs/test/lib/python3.7/site-packages (from requests>=2.25.0->kairos-yaml==0.0.1) (2.10)
Building wheels for collected packages: kairos-yaml
  Building wheel for kairos-yaml (PEP 517) ... done
  Created wheel for kairos-yaml: filename=kairos_yaml-0.0.1-py3-none-any.whl size=26318 sha256=d6b869b39510c3586651d900e3d2653e2368f866505064f730f1e24cb8957d19
  Stored in directory: /Users/ahedges/Library/Caches/pip/wheels/29/0b/6b/ca2ec5bbc2e68219fad5de816e270c6bbe7a35772fa192e9ef
Successfully built kairos-yaml
Installing collected packages: kairos-yaml
  Attempting uninstall: kairos-yaml
    Found existing installation: kairos-yaml 0.0.1
    Uninstalling kairos-yaml-0.0.1:
      Successfully uninstalled kairos-yaml-0.0.1
Successfully installed kairos-yaml-0.0.1

Additional information

I have noticed two similar issues, but they appear to have different problems:

• Unlike pip3 not upgrading a dependency #6457, I do not get a message about Requirement already satisfied for the package installed from Git. In addition, --upgrade-strategy eager does not help.

• Unlike pip does not reinstall an already installed package from a different direct url requirement #7678, using the new resolver does not help. In addition, running from their tags as well as their commits work properly. This leads me to believe that the new resolver works properly with commits, as long as the package versions are different. All the following commands execute properly:

  pip install --upgrade git+https://github.com/pypa/[email protected]
  pip install --upgrade git+https://github.com/pypa/[email protected]
  pip install --upgrade git+https://github.com/pypa/packaging.git@47d40f640fddb7c97b01315419b6a1421d2dedbb  # packaging=20.0
  pip install --upgrade git+https://github.com/pypa/packaging.git@71b46f5a4ad9275b67c4b4eae873930f2d604aa9  # packaging=20.1

Here is my output from pipdeptree:

kairos-yaml==0.0.1
  - lazy-object-proxy [required: >=1.4.3, installed: 1.4.3]
  - pydantic [required: >=1.7.2, installed: 1.7.3]
  - PyYAML [required: >=5.3.1, installed: 5.3.1]
  - requests [required: >=2.25.0, installed: 2.25.0]
    - certifi [required: >=2017.4.17, installed: 2020.11.8]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.2]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.3]
wheel==0.36.0

[uranusjr]

#5780 (which is also mentioned in #7678) contains more context to this. In short, package managers are generally allowed to treat packages with the same name+version combination as equivalent since it provides important optimisation oppertunities. The legacy resolver does not have many of those optimisations, and combining with the fact it not solve dependencies well, presents misconceptions users can end up relying on unintentionally.

Note that #5780 also provides a more concrete proposal to straighten up all the scenarios and provide usages for all user expectations. Feel free to comment on whther you feel it would solve the situation, or still misses some things that you need. I really want to get this right to avoid even more troubles in the future.

Closing this as a duplicate to merge the concersation in one place.

[brainwane]

Also, @aphedges, thank you for the detailed bug report!