Adding a direct URL dependency on setup.py breaks versioning, cached wheel is always used

[reuben]

Environment

• pip version: pip 20.0.2
• Python version: Python 3.6.10
• OS: macOS 10.15

Description
Situation: project TOOL version 0.0.1 depends on project LIBRARY version 0.0.1. Project LIBRARY is not published on PyPI, so project TOOL uses a direct URL dependency in install_requires, pointing to a GitHub tarball:

  # ...
  install_requires = ['LIBRARY @ https://github.com/user/library/tarball/some_branch#egg=0.0.1'],
  # ...

A user installs project TOOL for the first time from a wheel published by the project TOOL maintainers, using pip install TOOL-0.0.1-py3-none-any.whl. pip fetches the tarball from GitHub, saves it into its cache, and the install works fine.

Project TOOL author wands to update its LIBRARY dependency to 0.0.2. But there's no version references anywhere, as direct URL dependencies don't support version info. Instead of being able to make a new wheel with correctly versioned dependencies, project TOOL author must instruct all users to manually update to the new version of project LIBRARY.

Expected behavior
Pip does not cache direct URL dependencies since they don't have version info and therefore there's no cache invalidation criteria.

How to Reproduce
I don't have simple examples to share, but I guess I'll write them over the weekend to make it easier to reproduce this.

Output

In this log I'm installing (in a fresh virtual environment) TOOL==0.0.1, which depends on LIBRARY==0.0.1. Then, I update LIBRARY to 0.0.2, by pushing new commits to the same branch. I expect new installs to pick up the new version, but yet, when I install TOOL==0.0.2, it picks up the old version of LIBRARY from the cache:

$ pip install https://.../TOOL-0.0.1-py3-none-any.whl
Collecting TTS==0.0.1+487a431
  Downloading https://.../TOOL-0.0.1-py3-none-any.whl (332.7 MB)
     |████████████████████████████████| 332.7 MB 1.5 MB/s
Collecting LIBRARY@ https://github.com/some/repo/tarball/some_branch
  Downloading https://github.com/some/repo/tarball/some_branch
     | 51 kB 655 kB/s
Building wheels for collected packages: LIBRARY
  Building wheel for LIBRARY (setup.py) ... done
  Created wheel for LIBRARY: filename=LIBRARY-0.0.1-py3-none-any.whl size=42564 sha256=4a2853d8de9a33f59af7e5e6e8bf8a98e27c3d320603e5586d2c74fc3dffda11
  Stored in directory: /Users/reubenmorais/Library/Caches/pip/wheels/92/ab/20/81122180a9004f04f8566f6858a01ff0f6b1e32b58d7c82509
Successfully built LIBRARY
Installing collected packages: LIBRARY, TOOL
Successfully installed TOOL-0.0.1 LIBRARY-0.0.1

$ pip install TOOL-0.0.2-py3-none-any.whl
Processing TOOL-0.0.2-py3-none-any.whl
Requirement already satisfied: LIBRARY@ https://github.com/some/repo/tarball/some_branch from https://github.com/some/repo/tarball/some_branch in ~/.local/share/virtualenvs/test-EYXJdgxr/lib/python3.6/site-packages (from TOOL==0.0.1) (0.0.1)
Successfully installed TTS-0.0.2

[sbidoul]

@reuben your issue is probably not related to caching. I think you are being surprised, as others, by the default pip behavior with PEP 440 direct urls. Pip considers LIBRARY is already installed (Requirement already satisfied: LIBRARY@...) instead of upgrading it by default. Using pip install --upgrade TOOL-0.0.2-py3-none-any.whl may do what you expect. See also #7678, #5780.

That said pip does indeed cache LIBRARY-0.0.1-py3-none-any.whl in your case, and this is probably due to the URL containing something that looks like an archive with a version number in it's name.

The reproducible example you are proposing to write will certainly help clarifying this.

[reuben]

@reuben your issue is probably not related to caching. I think you are being surprised, as others, by the default pip behavior with PEP 440 direct urls. Pip considers LIBRARY is already installed (Requirement already satisfied: LIBRARY@...) instead of upgrading it by default. Using pip install --upgrade TOOL-0.0.2-py3-none-any.whl may do what you expect.

I'm explicitly installing a newer version of TOOL, so shouldn't that be an upgrade operation automatically? pip knows TOOL-0.0.1 is installed, and upgrading a package should also entail upgrading its dependencies, no?

See also #7363

Wrong issue/PR number?

That said pip does indeed cache LIBRARY-0.0.1-py3-none-any.whl in your case, and this is probably due to the URL containing something that looks like an archive with a version number in it's name.

Hm, the URL is a GitHub tarball URL from a branch, pip downloads the source and builds the wheel using setup.py. Maybe if I remove the version field from LIBRARY it won't be cached? I'll try to get that working, that'd be a suitable workaround solution I guess.

The reproducible example you are proposing to write will certainly help clarifying this.

Working on it.

[reuben]

I just tested with pip install --upgrade and that also doesn't upgrade the library. I'm going to try removing the version info from the library now.

[sbidoul]

Wrong issue/PR number?

Yes, sorry, I meant #7678 (#5780 is also similar)

[reuben]

Hm, this is weird, I definitely do not see the same behavior as #7678. I made TOOL-0.0.1 depend on LIBRARY-0.0.1:

    install_requires=[
        'LIBRARY@ https://github.com/reuben/pip_cache_library/tarball/[email protected]',
    ],

I built a TOOL-0.0.1 wheel, and installed it. pip downloaded the LIBRARY-0.0.1 source and cached a wheel for it. Then I bumped the TOOL version number, and changed the dependency to LIBRARY-0.0.2:

    install_requires=[
        'LIBRARY@ https://github.com/reuben/pip_cache_library/tarball/[email protected]',
    ],

Again I built the new wheel, installed it with pip install --upgrade TOOL-0.0.2-py3-none-any.whl, and it says:

Processing ./tool/dist/TOOL-0.0.2-py3-none-any.whl
Requirement already satisfied, skipping upgrade: LIBRARY@ https://github.com/reuben/pip_cache_library/tarball/[email protected] from https://github.com/reuben/pip_cache_library/tarball/[email protected] in /Users/reubenmorais/.local/share/virtualenvs/pip_url_cache-m4x0xJZl/lib/python3.8/site-packages (from TOOL==0.0.1) (0.0.1)

Basically it looks like the cache key is simply the library name, rather than name + version, and so I can't get it to upgrade the library in any way, other than manually instructing pip with pip install --upgrade "LIBRARY@ https://...".

The fact that the last command works but installing the wheel doesn't makes me suspect this logic is implemented twice? Maybe the bug I'm seeing is actually in setuptools, or the wheel package? I don't know anything about Python packaging internals so it's hard for me to know.

[reuben]

Ah, nevermind, what I'm seeing is exactly what's described in #5780.

[sbidoul]

Yep, same as #5780 and somewhat different from #7678 (TIL that pip behaves differently for top level requirements and dependencies in that respect). This is not related to wheel caching, though.

[reuben]

It's even more different than I thought. On the command line, this doesn't work:

$ pip install "LIBRARY@ https://github.com/reuben/pip_cache_library/tarball/[email protected]"

It requests the URL including the "@0.0.2" part, which is a 404. But in install_requires that syntax works fine.

[sbidoul]

I've just tested this setup.py:

from setuptools import setup

setup(
    name="TOOL",
    version="0.0.1",
    install_requires=[
        "LIBRARY@ https://github.com/reuben/pip_cache_library/tarball/[email protected]",
    ],
)

pip install . correctly says ERROR: HTTP error 404 while getting https://github.com/reuben/pip_cache_library/tarball/[email protected].

[reuben]

I've been testing by building and installing a wheel (because that's how I distribute TOOL to users). So, that same setup call, but python setup.py bdist_wheel and then pip install the_wheel.

[sbidoul]

I still can't reproduce:

$ python setup.py bdist_wheel
$ pip install dist/TOOL-0.0.1-py3-none-any.whl                                                                                                                        Processing ./dist/TOOL-0.0.1-py3-none-any.whl
Collecting LIBRARY@ https://github.com/reuben/pip_cache_library/tarball/[email protected]
  ERROR: HTTP error 404 while getting https://github.com/reuben/pip_cache_library/tarball/[email protected]

[reuben]

Sorry, I think I got tricked by the very bug I'm reporting here. The first time I installed it, the dependency URL didn't have the "@0.0.1" in it, and then for subsequent installs, because of this bug, pip never tried to fetch the URL at all, so I wasn't actually testing the new URL.

[sbidoul]

@reuben shall we close this as a duplicate of #5780?

[reuben]

Sure, that works for me.