Add support for URL dependencies (at least for wheels)

[charliermarsh]

• VCS:
  • Start with just git+
  • The URL must point to a source distribution
  • If the URL ends with a hash, we can use that as the cache key
  • If the URL ends with anything else, we need to resolve it to a commit, then use that as the cache key (and we need to do this lookup every time)
• URL dependencies can be a source distribution or a wheel
  • For URL dependencies, we should cache based on the HTTP semantics
  • Empirically, it seems that the URL must end in .whl to be considered a wheel; otherwise, it’s considered a source distribution
  • We could even start by not caching anything here

[charliermarsh]

I need to look at how some other tools handle this (e.g., Bun, pip, Poetry).

[charliermarsh]

oven-sh/bun#1875 (comment)

[charliermarsh]

pypa/pip#10075

[charliermarsh]

pypa/pip#11164

[charliermarsh]

Lots of good discussion in here: pypa/pip#10564 (comment)

[charliermarsh]

Gonna propose something here...

[konstin]

Unlike pip, we should remember the url contents we installed (e.g. through an etag or another http caching mechanism), check the url every time and reinstall if they changed. For locked requirements.txt, we would need hashes.

[charliermarsh]

Yeah we need a clear mechanism for this. But the pip issues also have a lot of discussion around what happens when you change from a URL dependency to a version dependency, etc.

[charliermarsh]

Probably the biggest remaining "feature" (with the rest of the milestone being largely focused on testing, performance, and polish).

[charliermarsh]

Another challenge here is that we need to fetch and build the distribution in order to know the version.

[charliermarsh]

We could consider attempting to parse the version from the URL, but it won't always be sufficient (and it could even be wrong).

[charliermarsh]

A few other considerations:

• If there's a URL dependency that satisfies a version, and another package depends on that version in a non-URL capacity, we probably need to allow that. For example, if a user points to a pytorch wheel via URL, and some other package depends on pytorch == 1.0.0, that should be allowed as long as the version at the URL matches the requested version.
• If there are two conflicting URL dependencies, we need to reject those, even if they resolve to the same version.

[charliermarsh]

Initial version is closed by #251.