[pypi] Support pulling dependencies using direct url specifiers without using an index

[mering]

🐞 bug report

Affected Rule

Module extension:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(...)

Is this a regression?

Not sure.

Description

The repository rule for pip dependencies doesn't use repository_ctx.download(). This breaks usages of --experimental_downloader_config for example to use a transparent dependency mirror.

🔬 Minimal Reproduction

https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_library.bzl#L245-L256

https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_installer/wheel_installer.py#L168-L190

🔥 Exception or Error

Stack trace prints to the code linked above.

🌍 Your Environment

Operating System:

Linux

Output of bazel version:

7.0.2

Rules_python version:

0.37.2

Anything else relevant?

[aignas]

Please see the documentation here: https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#bazel-downloader-and-multi-platform-wheel-hub-repository.

This currently has a limitation that it needs hashes for all of the files in the requirements.txt. The stabilization of this feature is tracked by #260.

[mering]

@aignas How can I use experimental_index_url in combination with dependencies without an index like the following:

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz
tensorflow_graphics @ git+https://github.com/tensorflow/graphics@ca63cf788978db0bde2db4ce488e102c455da2e7

[aignas]

The support for direct references has not been added yet, I'll reopen this issue and rename it to scope it to supporting it, but I am not sure when I can work on it.

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz could be easy enough. Outline of the changes:

• Update the https://github.com/bazelbuild/rules_python/blob/main/python/private/pypi/index_sources.bzl file to support extracting URLs from the requirements file
• Change the code that calls the index to not call it for requirement lines that refer to URLs
• Pipe the url and the filename to the whl_library respective attributes.

For the second one it may be harder - I am not sure how to handle git repos yet, however, with bazel 7.4 the extension can pass a labels within the extension more easily, so one could look into how to do that.

For now the workaround would be to build the wheels from sdists outside bazel and host them in some PyPI compatible artifact mirror and then use experimental_index_url flag to enable the usage of the bazel-downloader.

[aignas]

As @mering suggests the git_repository archive is downloading things by shelling out to the git executable source. I am not entirely sure that this is something that we can do easily because duplicating calling git should be out of scope of rules_python.

However, we should support git_repository when pulling sources for sdist building, which we want to officially support via build actions (#2410). Ideally we could just use the http_archive or git_repository to get the sources.

I'd like to mark this as done and add a note to the sdist building ticket for git sources support. Does that sound good to you, @mering?

[mering]

@aignas thanks for fixing the direct download case 1. When we applied this to our code base, we noted that it still tries to reach out to an index-like URL. Could this be skipped in that case?

I am fine with continuing the conversation about the git repository in the other issue.